What's going on, everybody? Welcome to my complete data analyst boot camp here on YouTube. Now, this data analyst boot camp, which is almost 24 hours long, is one of the best free resources in the entire world. There's no other YouTube channel, there's no platform, there's no place that you're going to get as much value for free than this video that you're watching right now.

In this boot camp, we're going to cover a ton of different things. We're going to be looking at MySQL, starting from beginner to advanced. We move on to Excel, Tableau, Power BI, Python, Pandas, and then we'll go into some cloud platforms like AWS and Azure. Throughout the boot camp, we work on a ton of projects, and these projects are meant to help solidify your skills as well as help you build a portfolio of projects to show to potential employers.

We aren't just looking at technical skills in this boot camp; we go into a lot of other things. So we talk about interviewing, we talk about how to create a good LinkedIn profile, how to reach out to recruiters, how to build a portfolio website for free. All of this is in this video, and so everything is here in one video, just for you.

If you haven't seen it already, I have a data analyst boot camp playlist where you can go and individually watch each of these videos. But I combined all of it into one massive video because some people really just like coming back to one video, not having to go and find their specific place that they're in, and so I wanted to do that for you guys. All the videos that you are seeing are videos that I've created over the past 5 years. This is 5 years worth of content, just distilled into one really big boot camp, and so I'm so excited for you to get started on this boot camp because I know there's so much that you're going to learn.

Now, before we start the boot camp, I want to mention my data analyst learning platform called Analyst Builder. Analyst Builder is like this data analyst boot camp, but on steroids, because I go about 50% more in depth into all these technical skills. We build more advanced projects, and we have a questions page; we can go in practice SQL in Python and R. It's really good for practicing for your technical interviews; it has all of my full courses in it. And so, if you really like the MySQL series here in the boot camp, then you can go and take the full MySQL course, and it's just going to be a lot more in-depth and have some more advanced projects. We also have a data analyst road map, which is my suggested path for learning on the platform. Throughout the courses, there's lots of practice problems and challenges as you go to make sure that you really know the skills by the time you finish the road map. It's genuinely the best platform to learn data analytics in the entire world, and we've made it more affordable than almost any other platform, so be sure to check it out. And if you are interested in it, I'll leave a discount code below in the description.

Now, one last thing that I have to mention before we start the boot camp is that there are a ton of data sets and a ton of data that we use throughout this entire boot camp. I have an entire GitHub dedicated with a bunch of different repositories that have all of the data sets in it. And so what you need to do is, as you're going through the course, you have to download the corresponding data set to make sure you can follow along with each lesson.

So, with all that being said, let's get started on our data analyst boot camp. We're going to start at the very beginning, assuming you haven't started this process at all of becoming a data analyst. If you already have, you can kind of find, identify where you are in this process, and then go from there.

Now, before we dive into everything, I want to warn you: I will be mentioning my own channel a lot in this video. I have videos and playlists on just about every single topic that we're going to be talking about today. I'll have all the links to those videos in the description so you can dive into those topics more in depth. So I hope that's okay, and it's all completely free. I've been building this out for the past 3 years, and honestly, you can probably get 90% of the way to learning everything you need for data analytics just on my channel.

So now that I've warned you, let's jump to number one, and that is learn the data analyst skills. Now, there are literally a hundred different things that you can learn for data analytics. You can learn things like Alteryx or a cloud platform or different programming languages, but there are some core skills that I recommend you start out with before kind of branching into some of those other skills.

The number one skill that I always recommend people start with is SQL. SQL is just one of those fundamental skills; I think everybody should learn, even if you don't use SQL, you'll use some variation of SQL. If your company has a large enough data set, SQL is used to actually query and retrieve data from a database. So if your company collects data, which every company does, they're going to put it somewhere to store it; it's usually stored in a database, and SQL's how you get that data from the database. I think SQL's also fairly easy to learn, which makes it really good when you're just starting out. I have several playlists dedicated to SQL, starting from beginner all the way to advanced, and you can learn all of that for free.

One other reason why I think you should learn SQL first is that a lot of companies interview, or have a technical interview during the interview process, on SQL. That's something that really caught me off guard when I was first starting out because I thought it was going to be more behavioral; I didn't even know what a technical interview was. So knowing SQL actually became a really important part of interviewing and getting a job as a data analyst.

The second skill that I would learn is a business intelligence tool like Tableau or Power BI. Now, there are a ton of different BI tools; I can literally name 10 off the top of my head that I've used throughout my career. But what I will say is that learning something like Tableau or Power BI is pretty transferable to almost all those other BI tools; they're all fairly similar in how they do things and how they show and display the data. You most likely won't have a technical interview asking you about Tableau or Power BI, like to build something for them; that usually does not happen. But the combination of SQL, where you can query your data, and then taking that data to build something, that is a really, really great combination to learn right away. I have entire series on both Tableau and Power BI, with projects on my channel.

The third skill that I would learn is Excel. Now, most people have used Excel; they know what Excel is and how it's used, but it can be used a little bit differently for a data analyst. For example, in Excel, a lot of people haven't cleaned data in Excel or built charts and graphs using Excel, and those are things that a data analyst would probably do. Excel is also just a fundamental skill that every company is going to expect you to know. So, so I have an entire playlist dedicated to Excel to actually walk you through how to use it for data analysis.

The fourth skill that I recommend you learn is Python. Now, a lot of people will have Python higher up on their list; they only use Python; they don't use SQL or a BI tool; they just do everything in Python. Now, Python is a fantastic tool; you can use it to manipulate your data, to create data visualizations, and a ton more, like web scraping and regular expressions and a hundred different other things. But it can be kind of hard to learn; it took me a long time to really learn the basics very well. That's really the only reason why it is farther back. I feel like SQL and a BI tool are really easy to learn and really pack a big punch, whereas Python can be quite tough to learn, in my experience, and you may not use it as often as you would something like SQL or a BI tool. If you're interested in learning Python, I have an entire series dedicated to Python, as well as projects that you can build. Again, I warned you: there's going to be a lot of self-promotion in this video. I have videos on just about every single one of these topics.

The fifth and the last skill that I recommend you learning, and this is the only one that I don't have a series on yet—I will make those—is learning a cloud platform like AWS, Google Cloud Platform, or Azure. There's no denying that these platforms have played a huge impact in how we use data as a whole in the data analyst industry. They can be kind of tough to learn, though, if you aren't using it hands-on in an actual job. I think that learning a cloud platform is already something that most people should start working towards because, in the future, it's only going to become more prevalent.

Now, where can you go and actually learn all these skills that you need to become a data analyst? Well, the number one place I recommend, of course, is my channel. I have free tutorials on all these skills and a lot of other topics, and I think it's just a really great place to start. The next place that I recommend you looking at is Udemy. I recommend Udemy especially if you're just starting out because it's pretty cheap; you can buy an entire course, an entire SQL course, for $10 or $15, and they have courses on every single one of these skills. And I just recently made a video called "DIY Data Analyst's Curriculum Using Udemy for Under $75," so you can create an entire curriculum to learn all of these skills for under $75, which is just amazing.

The next place I'm going to recommend you look is Coursera. Now, Udemy is fantastic; they have really good instructors and good courses, but as a whole, I find that sometimes Coursera just has more professional or better content. Coursera is a bit more expensive, though; you're looking at $59 per month for all of their courses, or you can pay upfront an annual fee of $399. So again, it's just a lot more expensive. I moved to Coursera once I started having a data analyst job and had a bit more money, but when I was first starting out, I just couldn't afford it, so I went to Udemy, and it was a really great place to start. There's also places like DataCamp and DataQuest that kind of gamify learning, and they're more text-based. So all these other platforms—Udemy, Coursera, and me—they're all video-based, but if you like reading, DataCamp and DataQuest are a lot more text where you can learn it by reading it and doing it.

After you learn all of these skills, the next thing that I recommend you do is actually build projects with those skills. Now, what does building a project actually mean? It means taking a skill and then building something out of it that you can then show a potential employer. For example, if you went through and learned Tableau, you go and take a data set, and you could build a visualization and a dashboard in Tableau, and that would be a project. With these projects, you can build something called a portfolio, and I usually call it a portfolio website. A portfolio website is a website that you create where you store all of your projects, and then you can share that with, with recruiters and hiring managers so that they can see all of your work.

Now, do you absolutely need a portfolio to show employers? No, you don't, but it does help in two different ways. The first thing that it may do is actually help you land the interview. If you have a link on your resume and they click on it, they may see your skills and see your projects and be like, "Man, this person really knows what they're doing; this is exactly what we need." The second reason that I recommend building projects is because most likely during your interview you're going to get asked questions like, "How have you used SQL? How have you used Tableau?" And if you don't have any experience in that, you're just going to say, "Well, you know, I've taken courses to learn it." But with a project, you can be a lot more specific; you'll be able to say, "Well, I actually just built out this project in Tableau; I took the data and cleaned it in Excel, and then I put it in Tableau and built out this dashboard, and here are the insights that I found from this data set." It's just a much better answer, and as a hiring manager myself, I can tell you that it is definitely beneficial to build out these projects.

The next step that I recommend you take in becoming a data analyst is building a data analyst resume. The resume, to say the least, is extremely important; it's what's going to actually allow you to land an interview to potentially get a job. Now, if you were like me when I was first starting out, I had a resume; it just had nothing to do with data analytics. So how do you make a data analyst resume if you don't have any experience as a data analyst? Well, you are asking the perfect questions because the very first things that we talked about are what are going to go on your resume: those skills and those projects. If you have no experience or degree, like myself, who has a recreational therapy degree—if you have no background in this, it can be really daunting to kind of display that you know what you're doing and that a company should hire you. So what I usually recommend is, right beneath your contact at the top, you put your skills and your projects that you built out on your resume. Things like work experience and education should go on your resume as well, but just a little bit lower. You want them to see those things before they see that your last work experience was at Domino's and you have a degree in Marine Biology; it's just not relevant to data analysis, and if you put those things at the top, they're probably going to rule you out right away.

The fourth step to become a data analyst is actually applying. You have the skills, you have the projects, you have the resume; now you're ready to start applying for those data analyst jobs. Now, there's a lot of different opinions on how you need to go about applying for data analyst jobs, but I'll give you my take on it, and this has been the most successful for me in my career. The first thing that I want to mention is actually what I would not do, which is just blindly fly on Glassdoor, Monster, ZipRecruiter, and all these other platforms to just any data analyst job that you can find. Now, I'm not against this; I think you should do that, but I don't think that's the only thing that you should do because the chances of you getting a call back or actually hearing something back are extremely low. To really increase your chances of becoming a data analyst, I highly, highly, highly recommend working with a recruiter. A recruiter is literally someone who is there to help you find a job. Now, when I first started out, I didn't understand what a technical recruiter was at all; I was kind of nervous or scared to work with them, but it's actually pretty simple. A company has a position that they want to fill, and they don't want to spend hours and hours and hours to find someone to fill that position, so they hire a recruiter. A recruiter is going to go out and try to find someone to fill that position—AKA you. And so if you go and talk to that recruiter and they have a position that opens up, they will help you get that interview, and then if you get a job, let's say for $50,000, the company is going to pay that recruiter, let's say 10% of your salary, so they'll give them $5,000. So you don't actually lose or have anything to lose using a recruiter. You can reach out to recruiters in several ways, and I've done every variation, but I'll tell you my most successful way, which was using LinkedIn. There are tens of thousands of recruiters on LinkedIn. I made an entire video of how you can reach out to recruiters and what to say to recruiters on LinkedIn to help you land a job, so be sure to check out that video when you actually get to that point. But you can also just cold email and cold call these recruiting companies, but to me, it's just not as effective as reaching out directly on LinkedIn.

And this is just a bonus one: the last thing that you need to do is accept a job offer. So on step number four, after you apply to those jobs, you do actually have to go in, interview, and then get a job offer, which you will accept. I just thought I'd mention that just in case that was not super clear.

Now that was a lot of stuff; let's talk about time frames to actually complete all of these things. Now, doing all of these things from scratch is going to take a while, but let's break it down by each step and see how long I generally think it's going to take. Let's start, start with step number one, which is actually learning the skills. Now, just to be up front, this one probably is going to take the longest for most people. For most people to learn all of these skills, it's going to take around 3 to 4 months. Now, if you don't learn a cloud platform and Python, which are the last ones that I recommend, and you just focus on SQL, a BI tool, and Excel, I think you can do that in under 3 months. That is very dependent, though, on how much time you have to study. That time frame is more for someone who has several hours per day, maybe 3 hours in the end of a night after you go to work; that is someone who has quite a bit of time to dedicate to learning during their week. Of course, that time frame is going to take longer if you don't have as much time to dedicate to learning.

Now let's look at number two, which was creating projects and a portfolio of projects. From my experience, when you're first starting out, it takes a lot longer to actually create these projects; it can take one or two weeks per project. I usually recommend people doing three to five projects in their portfolio before they start applying, and since they can take anywhere from 1 to 2 weeks, you're looking at anywhere from 3 to 6 weeks.

The next step was to create a data analyst resume. Now, in my opinion, this one should take the shortest out of every single step here because you're really just kind of reformatting a resume or creating a resume; you're just adding skills, you're adding your projects, and then kind of reformatting it to make it look nice. This should hopefully take under a week, but if you use something like a professional service where they help you build a resume, it could take one to two weeks.

The two last steps, which kind of go hand in hand, are step four and five, which is actually applying for jobs and then landing a job. Now, this process can take as little as a month, or it can take as long as 6 months or a year; it really depends on how you're applying, where you're applying, and just the kind of luck that you're having with actually landing interviews. I've seen people who have never had any experience land a job within a month of starting to apply, and it's incredible; it's amazing, but it doesn't happen too often. You're usually looking at around 2 to 4 months on average to land your first data analyst job.

If you put all of those together and kind of average everything out, you're looking at around 6 months total for the entire process. Now, I don't want that to discourage you, okay? 2023 is a long year; you have a lot of time, and it doesn't have to take 6 months; you could do it faster; you could do it in 3 months and just prove me wrong. But if you are really focused and you are really driven to become a data analyst this year, I know that you can do it.

Now, to maybe boost your spirits and make you feel a little bit better, I didn't know any of these things when I first started out; I didn't have anyone telling me kind of a plan on what to do; I had to go out and figure all these things out by myself, and it took me almost a year to land my first real data analyst job. So, with all that being said, I hope that this video is helpful; I hope you now have a path on how to become a data analyst.

Analyst this year, and that my channel can be a big part of that. So thank you guys so much for watching. I really appreciate it. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

Today, we are going to be starting our MySQL tutorial [Music] series. Now, the entire MySQL series will be broken up into three smaller series: we'll have our beginner, our intermediate, and our advanced. This lesson is the very first lesson in the beginner series, where we're going to walk through all of the beginner, or the basics, of MySQL. Today, we're going to be setting everything up, so we'll be installing MySQL and then creating our database that we'll use to actually learn MySQL.

Now, before we get started, I wanted to let you know that I created three full MySQL courses over on analystbuilder.com. I built a crash course for MySQL for people who are going to be interviewing or taking technical interviews. I also created a full MySQL course that's going to cover everything from the basics all the way to the intermediate level. And then lastly, I created an advanced course that's going to teach you a lot of the more advanced things that an analyst would typically use. Those courses are going to go really in-depth, and they're going to have a lot of practice questions along the way, and we'll also have full guided projects in there as well. I'll have links in the description to all of those courses if you want to check those out.

Now, without further ado, let's jump on my screen and install MySQL and create our database. All right, so let's get started by downloading MySQL. We're going to come right over here to dev.mysql.com/downloads/installer, and I will have that link in the description so you don't have to write all that out. But you should be seeing this page right here. Now we have to select an operating system. I'm using a Windows machine, and if you aren't—if you're using Linux or Mac or something else—it should populate it for you. But if it doesn't, just select this drop-down and select your operating system. Next, we have two different downloads: we can install the MySQL Installer Community or MySQL Installer Web Community. This one is very small, but then you actually do have to download the installer; it just gets it from the web. This one—I'm going to download the actual installer; it's larger, but this is the one I'm going to do. So I'm going to go ahead and select download. It's going to ask me if I want to log in or create an account, and I don't; I'm going to say "No thanks, just start my download." I'm going to save this in this desktop folder; it doesn't really matter where you save it. We're going to save that, and it's going to download; it should be done in just a few seconds. I'm going to go ahead and click on it, and it's going to open it up when it's finished, and we should get the interface, or the UI, for the actual installation for MySQL.

So here is the MySQL installer, and the first thing we need to do is choose a setup type. Now we're going to keep the developer default, unless you really know what you're doing, and you can select the server only, the client only, full—which is literally everything MySQL has to offer—or custom. So we're going to keep this developer default, just installing the things that we kind of need. So let's go ahead and select next. And for whatever reason, on my computer, it's saying this path already exists. You probably won't get that, but I'm just going to go ahead and select next, and then I'll select yes. It keeps doing that; I can't explain why, but it keeps doing that for me, even though I've deleted it from my computer completely. Um, but it just remembers it somewhere in its memory.

Now, the next thing we need is to check requirements. Now I just have this one; it says I need to download this uh Visual Studio. I'm not going to do that, but on your screen you may have multiple, multiple requirements. Typically, you're looking at something like this: Microsoft Visual C++ Redistributable Package. What you need to do is download this; all you have to do is click download. Once you download and install that on your computer, and then we go back, all of those should be gone. That's the one that I see the most when I'm actually working with these requirements. I had to install it myself when I—this new laptop—so go ahead and install that if you need to. But if yours looks like mine, we don't need this Visual Studio for what we're going to do, so we're going to go ahead and select next. It is giving us a prompt that we haven't satisfied all the requirements, but that's okay; we're going to go ahead and select yes as well.

Now we're ready to install all of these things. These are all things that MySQL wants you to install. The most important are the server and the workbench, but it does not hurt to have all these other things as well. Some of these connectors are also important, so we're going to go ahead and execute this. This will take just a few minutes. I'll skip ahead uh when they're all done, but this should take just a few minutes, and then we'll continue on installing MySQL.

So everything just completed, and now we're going to select next. And now we need to actually configure our product. Now, really the only one that we actually need to configure is the server. The router says we need to configure it, and the samples and examples say we need to configure it as well, but really it's just the server. Let's go ahead and select next. Now we're not going to change anything for this type and networking, unless you know what you're doing with the port, the X protocol port. Uh, we're not going to change any of this; we'll go ahead and select next. Next thing we need to do is select an authentication method. I'm going to be using a password; I'm not going to be using the Legacy authentication method, so I'm just going to go ahead and create a password. Now, for you—and I keep getting this error, and I can't explain why—right here for you, you should be creating a password at the bottom. It's remembering my password somehow, and I—I really can't explain it—but I'm going to create my password, or check my password. Uh, this is one that I already created uh before I deleted it off my computer, but it's still there. Um, so it's saying my password is still good, but if you need to, you should be entering a password and then confirming your password and saving it, and then you should also be checking it as well. And then we're going to configure this as a MySQL server as a Windows service. I'm going to keep that checked. Uh, and we're going to start the MySQL server at system startup. I like that automatically being there; I don't want to mess with that, so I'm going to keep it as it has it. We're going to go ahead and select next. And the last thing we do is just need to execute this, and then everything we put in there is going to actually go. So let's run this and execute it, and that's just finished. So let's go ahead and select finish. Now it's saying "Configuration complete for the server," but we also need to configure these other two. Let's take a look at these really quickly. We're not going to do anything on this; it even says we really don't need to do this; we just need to click finish. "Configuration not needed." Next, we'll do samples and examples, and we can input our password, and all this is really going to—to do is put in some sample databases for us in our database, which if you want you definitely can do that. Uh, I just connected; that worked. I'm going to hit next and execute, and it's basically just going to put in a database or two, some sample ones for you to look at, and the configuration is complete. You don't have to do that one, but we'll see that in just a second. We're going to select next, and now the installation is completely done, and we can start MySQL Workbench after setup and start MySQL Shell after setup. Now I'm not going to do the shell, so I'm going to actually uncheck that, and we're going to select finish.

Now MySQL just popped up for us, and this is exactly what you should be seeing. Now there's a lot of things in MySQL to learn and know how to do. We're not going to be taking a look at all of that stuff today, but in future lessons we'll walk through a lot of these different things that kind of correlate with different lessons or things that we're working on in MySQL. The first thing that we're going to click on is right over here; this is our local instance. This is local to just our machine; it's not a connection to, you know, some other database on the cloud or anything like that; it's just our local instance. We're going to go ahead and click on this. So this is what you should be seeing right here. This is where we're going to actually write all of our SQL code, and I'll show you all this in just a second, but this is where we can actually create our database, and our database is going to go right over here on this left-hand side. This "silicon1" is actually a sample database; it has a bunch of tables and views, stored procedures, functions; it has all these things in here. If you want to go ahead and mess around with that, what we're about to do is—is create our own database that we're going to be using throughout this entire series—both the beginner, intermediate, and advanced—we'll use a lot of this, and sometimes we'll import some other ones for different use cases, but this will serve for most of what we're trying to do throughout this entire series.

Now what I'm going to do is I'm going to go ahead and I'm going to say "Open a SQL script file in a new query tab," and right here it opened up to a folder that I already created: this MySQL beginner series folder. Within it, we have this right here: the "Parks and Rec create_DB." Now, in order to get this, you just have to go to the GitHub and download this file; that's all you have to do. We're then going to open this file, so let's click on it; we're going to say "Open," and what you're now seeing is basically the query editor. This is where you can write your code. Now we're not importing a database; we're actually creating it by running code. Now, because this is the first lesson in the beginner series, I'm going to assume that you don't know a ton about MySQL. Really, all this is doing is creating the database name, and then we're inserting a few tables into that database, and then we're inserting data into those tables. So this is all of our data that will go into these tables that we create. We only have one, two, three different tables that we're going to be using. So all you have to do to run this is click this lightning button right up here. We're going to go ahead and execute this. If we come down to the bottom and we pull this up, this is our output. This says "six rows affected," and we have a bunch of other things like "create table," "create table," "insert," "insert," "create table," "insert into"—these things are all working perfectly. So now if we go ahead and click refresh in our schemas with this refresh button right here, this "Parks and Recreation" table is populated. If we go under the tables, we see all of these things. So now that we've actually created our database and our tables, that's really all we were trying to do in this lesson. But I just want to open up a table really quickly, show you what it looks like, show you how we can run code, and—and then in the next lesson we'll start actually learning how to query this data. So let's go up to "employee demographics"; we're going to right-click and select "Rows limit 1000." This is going to open up a new window right up here, and it's going to say "select everything from this database.this table employee demographics," and it ends with a semicolon. Now, right down here we have this output window; this is the actual data that sits in our table. We have columns right here: so "employee ID," "first name," "last name," "age," "gender," and "birth date," and then here are all of our employees on each row. So these are all separate rows: we have Leslie, Nope, Tom Haverford, and it goes on and on. So this is all of our data. The most important things to know when we're actually working with this—and I'm going to zoom in—is if we hover over this query right here and we run it, we can select this "execute," which is this lightning bolt with this I. We're going to execute this, and it's going to run this because we're highlighted over it. Now if we have two queries—let's say this one right here—but let's change it to "employee_salary." Let's say we want to query this table. So now if we highlight over this and we go up and select the lightning bolt with the I, now we're looking at a different table. But if we select—even if we're hovering over this—if we select this button, we're going to execute everything in this editor window. So let's run this, and now you can see at the bottom we have two outputs: the "employee demographics" and the "employee salary." So this button is going to run everything in this editor window, whereas if we select this lightning bolt with the I, we're doing everything that's just under where we have the cursor, where we have it highlighted.

The very last thing that I want to mention is that right over here you may have this up, and you probably don't want that. We're not going to do any SQL additions in this series; you can get rid of that by clicking this button right here. So starting in the next lesson, we'll have everything ready to go, and we will start learning the basics of MySQL. I hope you're able to follow along, get everything set up how we have it on this screen. I am super excited about this series; I just—I love SQL—so I'm really, really, really excited to get started on this with you guys. I will see you guys in the next lesson [Music].

Hello everybody. In this lesson, we're going to be learning about the SELECT statement in MySQL. The SELECT statement is used to work with columns and specify what columns you want to see in your output. The first thing that we need to do is open up a tab or an editor window. So let's come right up here to the left-hand side, and we're going to create a new tab, and I'm going to zoom in just a little. Now what we need to do is we need to select the actual table that we're going to be querying off of. If you remember from the very first—when we set everything up—we came over here and we right-clicked and did "select rows limit 1,000." We're not going to do that; we're going to actually write it out. So what we need to do to select that table, the "employee demographics" table, is we need to select everything; that's what the star means. The star means everything: all tables, all rows. Now we do have a limit on here; we have a limit to 1,000 rows. So if we had a table that had 50,000 rows, this limiter would be an issue; it would still limit it to 1,000 rows. We would have to change that to 2,000, 5,000, probably all the way up to 50,000 if we wanted to view everything. If we had, say, a million rows, we would need to come up here and say "don't limit," and it would give us a million rows. The reason they do this is mostly to keep the processing time low. If you have a million rows, it's going to take a long time for the output to actually appear. So let's come right back here. The next thing that we need to do is we need to say "select everything," and now we need to say where we're selecting it from. So we're going to come right down here, and we're going to say "from," and now we need to specify what table, and we're going to say "employee_demographics," and at the end we need a semicolon. Now, why do we need a semicolon? This is going to tell MySQL that this is the end of this query. So if you write another one down here, which we will in just a second, it'll be able to distinguish between the two queries. We're going to go ahead and we're going to run this, and we'll just use this "execute" right here instead of this one, and there we have our entire table. So we were able to get our table.

Now there is one thing that is potentially wrong, depending on what you're using it for, but what we didn't do is we did not specify the actual database before it; we only specify the table, and this works perfectly fine because if you look over here on this left-hand side, we have "Parks and Recreation"; it's in black, it's bold; that means that we're hitting off of this database. What's going to happen though if we come down here to the CIS database and we double-click on it? Now this database is highlighted, so now when we're selecting this table, we're trying to select this table from the CIS database. Let's go ahead and try this. If you notice, we have no output. Let's come right down here and pull this up; it's going to say "employees sis.employee_demographics doesn't exist," so it's assuming that we're highlighting this CIS database. That means we're trying to pull from that database. Now we can still have this highlighted and still select the correct database by saying "Parks_and_Recreation." And let me spell that right: dot. So now we're selecting everything from Parks_and_Recreation.employee_demographics. If we run this, we do get the correct output. That's just something to consider, especially when you're working with a lot of databases and a lot of tables. It's usually best practice to actually put the database in front of the table name, although throughout this lesson we probably won't be doing that every time since we're only going to be using this Parks and Recreation database. Let's go ahead and double-click this so we have this highlighted again, and let's click all of this; let's copy all of this. We're going to come down just a little bit right here.

Now, so far we've only selected everything, but we don't have to do that; we can actually just select one column if we would like to. For example, if we got rid of that star, we say "first_name," we're selecting the first name column from this table. If we highlight this query and we hit the execute button with the I, now we are only going to return in our output all of the first names. And we can add a lot more; let's actually look at all these. We can separate multiple columns with a comma, so we can do "first name, last_name," and then we could do "birth_date." So now we have three separate columns. Let's go ahead and run this, and now we have "first name," "last name," and "birth date" in our output. Now the way we just wrote it is all on one line, and that's perfectly acceptable because MySQL is going to read it the exact same as if we did it in a different format, as long as it's still in this order. But sometimes you'll see it like this, where it—select first_name, last_name, birth_date—all on different rows. Now there's a lot of different use cases for this, or reasons for this, but it typically can be easier to read. Also, if you're doing any type of functions or calculations in the select statement, it's easier to separate those out on its individual row. Now again, we won't always be doing this, but it does help sometimes if you're doing that; it just makes it easier to visualize. For example, if we added the age—so let's add age in here—let's run this. Let's say we were doing a calculation where we wanted to add, you know, 10 years to their age, so we'll say "age," and we'll actually create a new row for this, or new column; we'll do "age + 10." So now we can easily see that we're doing plus 10 here, and this is another thing that you can do in the select statement: things like calculations. So if we go up here and we run this, we'll now have an "age" column, but we're also have an "age + 10" column where it just adds 10 to the age, and we can at least visualize and really easily see this.

When we're doing these calculations, now something really important to know about any type of calculations, any math within MySQL, is that it follows the rules of PEMDAS. Now, PEMDAS is written like this: it's P M D A S. Now what I just did right here with this pound or this hashtag is actually create a comment, so this code isn't going to actually run, but it's just for note-taking or seeing things in your actual editor window. Welcome back to comments in just a second, but just wanted to explain what that was.

Now what PEMDAS is is the order of operations for arithmetic or math within MySQL. This stands for parentheses, exponent, multiplication, division, addition, and subtraction. So this is the order that these calculations are going to run in the execution engine that MySQL has. So if I do age + 10, we'll put that all in parenthesis, and then we come over here and we add time * 10, so we're doing plus 10 here and then a times 10 here. What's going to actually happen is it's going to say age + 10, so 44 + 10 = 54. Then we're multiplying time * 10. The parentheses executes first because it comes first in this order: parentheses, multiplication comes next because it's this one, and then anything else after that. If we did, you know, plus 10, could run this, and you'll notice that it still follows the logic: 10 was just added at the very end to all of these outputs.

Now let's go right back up here. Let's select everything again from this table. Let's pull up this table so we can see it a little better, and let's go down because the last thing that I want to show you is something called DISTINCT. Now this is really, really useful, and you use this a lot in MySQL. What DISTINCT is going to do is it's going to select only the unique values within a column. Let's go ahead and copy this employee demographics, bring it right down here. Let's say SELECT and let's do first_name. So now we're just selecting the first name. Let's come right down here; there we go. So now we're selecting just the first name from this column. Now these are all unique values, so if we come right here and we say DISTINCT, nothing should happen to this table because these are all unique values. Let's go ahead and run this. As you can see, the output looks exactly the same. But what if we were to do something like gender? So let's come here, let's do gender, let's run this. Keeps going down; I don't know why it's doing that. Um, but now we have male and female. Now these are not all unique; we have female, female, female, and female, and the rest are males. So there's only two unique values here. So if we come right here, we say DISTINCT gender, now there should only be two in the output: male and female. Let's go ahead and run this. So now we get male and female in our output.

Now this works perfectly in one column, but what happens if we have two columns? So let's do first_name, gender. Let's go and run this. Now the combination of first name and gender are no longer unique. Now Leslie and female are being grouped together, and it's taking the distinct between both of these columns. So when we're only working with gender, it's only looking at the one column for both male and female, so it reduces it down to the only two unique values. But because we added the first name, all of these values are unique, so therefore the name plus the gender combination is always going to be unique.

The very last thing that I want to show you in this lesson doesn't actually pertain to the SELECT statement, but I want to save this code. Let's say we wanted to update this or upload this into our GitHub or save this and send it to somebody. We can do that; we can save it by clicking this save button right here. I'm going to go ahead and click this, and now we're in our MySQL beginner series folder. I'm just going to save this, and I can save this as anything I want, so I'm going to say 2. select statement tutorial. So now when I save this, you'll notice that the name gets changed up here to 2. select statement tutorial. Let's exit out of this. I'm going to open up, and now I'm going to come here to the select statement tutorial. I'm going to open it, and now I have our code again, exactly as we had it written before. I just wanted to show that to you in case you wanted to save your code as you go throughout this series, because that's usually what I do when I'm working with this stuff or learning these things: like to save my code as I go along. So with that being said, that is the end of the SELECT statement. In the next lesson, we're going to be learning about the WHERE statement, or we can actually filter our data. [Music]

Hello everybody. In this lesson, we're going to be taking a look at the WHERE clause. The WHERE clause is used to help filter our records or our rows of data, whereas the SELECT statement is used to help filter or select our actual columns. So when we're using the WHERE clause, we're only going to return the rows that fulfill a specific condition. Let's take a look at exactly how this works. Let's say we come right up here; we're going to say WHERE, and let's go down with that one. Let's say WHERE, and now we need to specify what column we're about to create this condition for. So we're going to say first_name. So we're saying WHERE the first name, we'll say, is equal to, and let's do quotes and let's say Leslie. So we're saying the first name has to be equal to this value right here, which is Leslie. For Leslie, nope. If we run this, there's only going to be one row that's returned because Leslie is the only Leslie in this entire table. Now we just used an equal sign, and that's actually called a comparison operator, and there's a few other comparison operators that you can use. Let's take a look at some of these other ones. Let's pull this down right down here, and let's actually highlight the SELECT FROM, and we're going to run it with this one right here. It's going to only select everything from the whole table, so we didn't select that WHERE clause. Let's go right down here and let's look at this salary field. So I'm going to say WHERE the salary, and I'm going to do a different comparison operator called greater than. So when the salary is greater than 50,000. Now one thing I want to note before we actually run this is that right down here we have Tom Haford who makes exactly 50,000, and I think there's one more: Jerry Git, which also makes exactly 50,000. If we run this, you'll notice that both Tom and Jerry are not in this output, but in the salary field, everything is greater than 50,000. The reason for that is that Tom and Jerry made exactly 50,000. What we're saying right here is WHERE the salary is only greater than. If we want to include Tom and Jerry, we have to say greater than or equal to, and now we'll select 50,000 or above, whereas right here before, when you're doing just this, it was greater than 50; it didn't include the 50,000. Let's go ahead and include it and run this, and now you'll notice that Tom and Jerry were both included because they had exactly 50,000, and we said greater than or equal to. Now we can do the exact same thing but with less than, so we have less than 50,000, and now we only have two people who make less than 50,000: that's April and Andy. And if we say less than or equal to and we run that, now we include both Tom and Jerry who make exactly 50,000, so it's less than or equal to $50,000.

Now what we're going to do is head on over to a different table. We're going to do the demographics table; make sure I spell that right, and let's add our semicolon. Let's run this, and what we want to look at is the gender really quick. So we're going to say WHERE the gender is equal to; we'll do in quotes female, and if we run this, we get all the genders that are equal to female. But we do have something called the not equal to, and it looks like this: it's an exclamation point and an equal sign. This is going to say WHERE the gender is not equal to female. So if we run this, you'll notice that the gender is all male. Now, now so far we've worked with things like integers, which are numbers; we've worked with characters or strings like names, but there's a different type of data type as well. In here we have a date column for these birth dates. Now in the WHERE clause, we can also filter on birth dates. Let's come over here, and we to say birth_date. Let's say it's greater than, and within quotes we'll say 1985-01-01. This is kind of the standard default date format within MySQL, which is year, month, and day. If we go ahead and run this, we can also take all the people who are greater than or born greater than 1985, so all of these dates are greater than 1985.

Now the next thing that I want to take a look at is logical operators in the WHERE clause. So logical operators are things like AND, OR, and NOT. Now these are called, and let's add this: logical operators. So logical operators allow us to have different logic, and let's take a look at how this works exactly. Let's copy this down because we already have this one written out. We're saying WHERE the birth date is greater than 1985. We can also say WHERE the gender is equal to male. So we can say AND the gender is equal, and then we'll say male. So we're adding a different complexity or an additional conditional statement within our WHERE clause. Let's go ahead and run this. So now we're only selecting birth dates that are greater than 1985 and where the gender is equal to male. Only the rows that fulfill both of those are returned. Now the AND says both this and this have to be true, but we could change this; we could say OR. What this means is is either this one has to be true or this one has to be true in order for it to be returned. So let's go ahead and run this. You'll notice that Jerry Gurit was born much before 1985, but since he has a male gender, he is in our output. And we could also use the NOT operator by saying OR NOT gender equal to male. So now what this is saying is the birth date could be greater than 1985, or it could not be equal to male, which is female. So if you look at Leslie Nope, she was born before 1985, but because she is female, she is in the output.

Now, like we talked about in the last lesson, there is something called PEMDAS, and that actually applies to these logical operators as well. So if we run this entire table, let's go ahead and run this. If we're looking at this entire table, let's say we want to get someone very, very specific. Specific. Let's say we're going to do uh WHERE the first_name is equal to Leslie and their age has to be equal to 44. That's extremely specific, and we can actually just do it like this: we don't need quotes um for integers; we could just do the number if we'd like to. This is very specific; this is only one person, but if we put this in parentheses, we can add an OR over here. Here we could say OR the age is greater than; let's just do 55. Let's go ahead and run this, and then we'll take a look at it. So within these parentheses, we have an AND operator. What that means is both this condition has to be met and this condition has to be met, and that's only one person: that's Leslie Nope. But then outside of these parentheses, we have another conditional statement: OR the age is greater than 55. So what we're saying within these parentheses is that this is an isolated conditional statement within these parentheses. If this is true, then in our output it'll be returned, but then we have an OR condition, which says OR someone with the age of greater than 55 can also be in the output. So these parentheses can be really helpful when you're actually using it in the WHERE clause with these ANDs, ORs, and NOTs.

Now I want to take a look at just one more thing, and let's bring this down here and let's get rid of this entire thing. Now the last thing we're going to take a look at is a LIKE statement. Now the LIKE statement is super unique because we can look for specific patterns; we're not necessarily looking for an exact match. Like here, if we said WHERE first_name is equal to Jerry, if we're looking for Jerry, it has to be exactly Jerry. But if we take this out, say J, and then we run it, we get no output, but it has to be an exact match. But here's where the LIKE statement comes in because we can actually say LIKE Jer, and we can add two special sequences or special characters within our LIKE statement. So those special characters are the percent sign and the underscore. The percent sign means anything, and the underscore means a specific value. Let's see how that actually works. So what we're going to do is we're say LIKE J%, that's the first one in this LIKE statement. What this says is the first name is like starting with Jer, but then has anything after it; it doesn't matter what it is, as long as it has Jer at the very beginning, it will be returned. Let's go ahead and run this. Now the only person who starts with Jr is Jerry. But what if I took the J out of here? Now it's saying it starts with er, and that's not anybody. What we can do is we can add another percent at the beginning. This is going to say anything comes before, anything comes after; all we're looking for is er somewhere in their name. Let's go ahead and run this. There still is only one person, and that's Jerry. Now let's come up here and let's get rid of this and let's say we're looking for everyone's name who starts with an A. We can do that really easily by saying A%. All that says is it starts with A; we don't have a % before it, which would say this string just has to have an A somewhere in it. If we have it like this, this means an A has to come at the beginning. Let's go and run this. In our output, we have April and Andy.

Now let's take a look at the underscore. If we get rid of this percent sign and we do two underscores, one, two, this is going to say it starts with an A and then it has two characters after it: no more, no less. So if we run this, An is going to be the only person who's returned because she has an A and then two characters after it. Now if we want Andy, we can specify that by doing another underscore: that's one, two, three, and now Andy is the only one in our output. Now there was also April in there, but she had more than three characters. But we can actually get her in our output by doing a percent sign. So we can combine both the underscore and the percent sign, and this is going to say it starts with an A, has one, two, three characters, and then it can have anything after that, so it just has to have at least an A and have one, two, three characters after it. So let's run it. Now you can see April comes into here because she does have an A, the P, R, and I are the three next characters, but then we have a percent sign that allows that L to be in the output as well. Now we don't just have to do this with strings or text like like April and Andy; we could also do this with birth dates. For example, Andy's birth date is 1989. We could say WHERE the birth_date is LIKE; let's say we want to look at everyone who is 1989 or born in 1989. Let's go and run this, and Andy's the only person born in 1989. But again, we looked at the year at the very beginning, so that is how the LIKE statement works. It looks for a specific sequence within that column that you can search for, so it doesn't have to be an exact match as long as it has that specified sequence that you've put in there anywhere within that cell or that column. So that is everything that we're going to look at for the WHERE clause. In the next lesson, we're going to take a look at the GROUP BY and the ORDER BY within MySQL. [Music]

Hello everybody. In this lesson, we're going to be taking a look at GROUP BY and ORDER BY in MySQL. Now when you use the GROUP BY clause in MySQL, it's going to group together rows that have the same values in the specified column or columns that you're actually grouping on. Once you group those rows together, you can run something called an aggregate function on those rows. Let's see how this actually works. Let's go ahead and copy this right here; we'll bring that down, and let me go back up one. Let's go ahead and write gender right here. Now we want to group on this gender column, and we're going to say GROUP BY gender. Let's go ahead and run this; we'll see what we get. And so we have male and female. Now we could get the exact same output by saying SELECT DISTINCT gender from this table. What is GROUP BY doing that the gender actually isn't doing well? It's actually rolling up all of these values into these rows, so later when we run aggregate functions like average, min, max, we'll do it based off of these rows, and all those rows are rolled up into these two rows, and we'll see that in a little bit. Now what if I was to come up here and in this demographics we have a first_name. What would happen if I'm selecting the first name but I'm grouping by the gender? Let's go ahead and run this. If we come right down here, we pull this up, you can see that the select list is not in group by clause and contains nonaggregated columns. What this means is that when you are selecting a column, if it's not an aggregated column like say average of something, if we're not using the aggregate functions in the SELECT statement, it has to be in the GROUP BY. These have to match, so this gender has to match this GROUP BY if we're not performing an aggregate function on it. Let's go ahead and run this, and now it works properly.

Now let's go back up; let's run this query because I want to select everything again, again, but let's say we wanted to take a look at the average ages for gender. So what we're going to do is we're selecting gender; we're also grouping by gender, but what we're going to do is add a comma, and we'll say the average, that's AVG, that stands for average, and then we're going to put in here age. So now this right here is an aggregate function; this does not need to go in the GROUP BY. We're just grouping on the gender, and then we're performing this aggregate function or kind of a calculation based off of those grouped rows for gender. So let's go ahead and run this and take a look at the output. So what this is telling me is that for the males, all of the male rows that were grouped, the average age is 41.3, and for female, the average age is 38.5. So super quickly you can tell that the average age of females is lower than the average age of males. Now we'll take a look at aggregate functions more in just a little bit. Let's actually go to a different table. Let's come right down here; we're going to go to the salary table and just select everything for now. Let's go ahead and run this. Now what we're going to actually be grouping on is this occupation right here. Now there's a lot of unique values; it's um not as distinct as the gender, which only had two values. You'll notice we do have a few that are the same; we have ones like office manager. So when we come up here, we going to say occupation, and of course we need to group by the occupation as well. Now let's run this. You'll notice that office manager only has one row. Let's say we also want to group on the salary; let's say salary. Now we can group on multiple, so we're going to say salary like this. So we're grouping on the occupation as well as the salary. Now let's run this. You'll notice that we have two rows for office manager. Now this is because

This salary and this salary for those two employees are different. We have 50,000 and 60,000. For this, I just wanted to demonstrate that if these had both been 50,000, there would only be one row: Office Manager 50,000. But because this is a unique value, different than 50,000, they have their own individual rows, which we would then perform our aggregate calculations on.

Let's go and get rid of that because we will not be using that anymore. I just wanted to demonstrate it really quickly. So before we were looking at gender and average age, and we're also grouping on the gender. We can perform other aggregate functions as well. Let's take a look at some of those. We could look at the max age as well. The max is going to show us the highest value within each of those groupings. So we have male and female; the max age for those, for the male is 61, and the highest age for the female is 46. We can do the exact same thing; exactly, we can say Min, or the exact opposite thing, we can say the minimum minimum age. So this is going to be the lowest for both the male and the female. Go and run this. Now we have female and male, and the minimum age is 29 and 34.

And there is one last one that I want to show you, which is Count. We're going to do Count now. Count is going to count the actual rows within this age column. So if we run this, you'll see that we have four females per count and we have seven males. It's just telling us a count of how many values is in this column when we're actually grouping on the gender. So that's how we can use groupby to actually roll up and group all of these similar values within a column or columns and perform our aggregate functions on them.

Now let's come down here, and what we're going to take a look at is order by. So we're going to say order by. Now let's actually pull in this demographics table right here. We're just going to say select everything and run this really quickly after we had a semicolon. So order by; order by is going to actually sort the result set in either ascending or descending order. Let's take a look at how this works. At the very end, we could say order by, and we could order by the first_name column. So we're going to take this column; we're going to order all of our rows based off of this one column. Let's go ahead and run this. So it's going to do it based off ascending, or which means smallest to largest. Now this is a text column or a character column, so we do it A to Z; so Andy and April all the way down to Tom. Now, by default, this is in ASC order, ascending order, and if we run this, it's going to be the exact same output, but we can change this to do it the opposite, highest to lowest, or Z to A, by doing descending. So now if we run this, you'll see that goes Tom all the way down to Andy.

Now let's take a look at ordering on something like gender and age, because we can do both at the same time. So let's order by the gender first. Let's go ahead and run this, and you'll see that all the females are grouped together, and then all the males are grouped together, because that's just the order in which it is. But we can do an additional column; we could also do it based off of the age. Let's go ahead and run this. So now, within the female, since that came first in our order by, we're ordering by the gender, and then we're also ordering by the age after we've ordered by the gender. So now it's 29 all the way up to 46, then 34 for males all the way up to 61. Now we can change this just for the age. Let's say we want to do age descending. So gender will stay the same in ascending order, but now age will be in descending order. Let's go ahead and run this. Now female and male stay the same, but now it starts at the highest down to the lowest.

Now this is something that I would absolutely do in real life, except sometimes you can make mistakes, and sometimes you do the wrong column first. Let's do age, and then we'll do gender. Now if we run this, the gender is not going to be used at all, and this is because there are no unique values that are going to be on the same row. So notice all these values are completely unique, so the gender never is actually used to order anything on because if there were things like 34, 34, 34, 34, these would be ordered based off of the gender, but since there's no unique fields, this is really pretty useless. That's why the order of the order by, or the columns that you place in the order by, are actually quite important.

Now the last thing that I want to show you, and I'll just go back to gender and age, is that you don't actually have to use the column names. We can use the column position. Now I will preface this by saying I don't recommend doing this, but I sometimes do it in a pinch for just a quick query, um, if I know the column position and I don't want to write out the whole name. So sometimes I do it, although it's not best practice. But let's take a look at it. So gender is the 1, 2, 3, 4, fifth column. I'm going to replace this with five, and age is the 1, 2, 3, 4 column. So these are the positions of the fields, but not the names of them. If we run it, we're going to get the exact same output, because these represent these columns appropriately. But again, I just don't recommend it; it's kind of a slippery slope that I've fallen down myself, uh, many times. And when you get to more advanced SQL and you're creating things like stored procedures and triggers and all these things, this can actually cause a lot of issues. If you were to add any columns or remove any columns, then you'd be ordering by the wrong column. Because let's say this last name got removed; we didn't want it for some reason, then the gender is 1, 2, 3, 4. Now we're ordering on the wrong column, and that would be a big mistake. So just by best practice, it is better to do gender, um, age. But I just wanted to show you that in case you want to be like me and kind of go down the wrong path, uh, so that is everything we are going to take a look at with Group by and order by.

In the next lesson, we're going to be taking a look at having versus where.

Hello everybody. In this lesson, we're going to take a look at the difference between HAVING and WHERE. Now in the last lesson we looked at GROUP BY and ORDER BY. The most obvious thing to do would be to come right here and say WHERE, and we're going to say this column, which is actually named this, we'll say WHERE the average age, let's say, is greater than 40, which would only be the males. So let's go ahead and run this, and as you can see we're not getting any output. Let's bring this up and take a look at the error. It says invalid use of the GROUP BY function. What's actually happening is something to do with this GROUP BY gender right here. When we're selecting gender and then we're performing an aggregate function, this occurs only after the GROUP BY actually groups those rows together. So when we're trying to filter based off of this column right here of average age, it really hasn't been created yet because this GROUP BY hasn't happened. That's where the HAVING clause comes into play. So let's go ahead, and what we're going to do is we're going to get rid of this; we're going to come right down here, and instead of WHERE, we're going to say HAVING. Now HAVING was specifically created for this exact example; it comes right after GROUP BY, and after GROUP BY we can filter based off of these aggregate functions. So now if we run this, we're going to get an output that only has where the average age is greater than 40.

Now let's take a look at just one more example, and I'm going to show you how you can use both in one query. So instead of demographics, let's look at the salary table and let's run it. Now in this salary table we have this occupation, and remember we have this Office Manager that happens twice, and this is going to be our main example. So we're going to say occupation, and then we'll say the average salary. Now we'll need to come down here and we'll say GROUP BY. Now we're going to say occupation. So this should look pretty similar because right here we have our Office Manager, and one of the Office Managers made 50, one of the Office Managers made 60, so the average is 55,000. Now I can use the WHERE by saying WHERE, then I'll say occupation LIKE, and let's see people who are managers, so I'll say %manager%, and close that quote. So they're like a manager, and then I want to see where a manager makes more than, let's say, 75,000. So I won't actually say WHERE; I'm going to say HAVING an average salary, and I need to add a space there: HAVING an average salary greater than, let's say, 75,000. And let's run this. So now I filtered at the row level right here in the WHERE clause, but then down here I filtered at the aggregate function level. This HAVING is only going to work for aggregated functions after the GROUP BY actually runs. So that is the difference between the HAVING clause and the WHERE clause. The WHERE clause you're most likely going to use a lot more, but if you do want to filter on those aggregated function columns, you have to use the HAVING clause. I hope that that was really helpful, and in the next lesson we're looking at our very last lesson in our beginner series; we're going to look at LIMIT and aliasing.

Hello everybody. In this lesson, we're going to be taking a look at LIMIT and aliasing. LIMIT is just going to specify how many rows you want in your output. If we take this table for example, if we come right here and we say LIMIT, let's do three. If we run this, it's only going to take the top three that we have. Let's go and run this. As you can see, we have employee 1, 3, and 4: Leslie, Tom, and April. Now this seems super straightforward, really really easy, but it can be combined with ORDER BY to actually be really powerful. For example, let's say we wanted to take the three oldest employees. All we'd have to do is come right under here; we say ORDER BY, and we'll order by the age in descending order. So we're going to order on age descending, and it's going to take the top three. So if we run this, and very quickly, we have the top three oldest people in this table.

Now there is one additional parameter that we can use in LIMIT, and all we have to do to access it is have a comma here. Now what this is going to do, and I'll put a one here, what this is going to do is it's now going to say we're going to start at position three, and then we're going to go one row after it. Now I actually want to take one of these people, so let's start at position two and select the next one after it, which should be Leslie. Nope. So we're going to start at position two, and we're going to select the one right after it. So we're going to start at position two, and then one means we're taking the next one row. Let's go ahead and run this, and as you can see we got Leslie Nope in our output.

Now let's come right down here; we are going to now look at aliasing. Now aliasing is just a way to change the name of the column for the most part, and it can also be used in joins, but we're going to take a look at joins, or aliasing joins, in the intermediate series. In a previous lesson we looked at a GROUP BY that looked like this: we selected gender, then we said FROM, I believe it was Employee_demographics, then we said GROUP BY gender, and we also had the average, and I think it was age there we go, and we'll add our semicolon. Let's go ahead and run this. In our output we have gender as our gender column, the same as the column name, but then average age is average age. And so if we want to actually do something like a HAVING where we say HAVING the average age, let's say, greater than 40, like we had it, we have to actually use this aggregate function in our HAVING clause, and we don't want to always have to do that. We can actually change the name of this column and subsequently use it throughout our query with that alias name. So I'm going to say AS, and that's the keyword to actually change it; we'll say AS, and we'll do average_age. So now we've changed this name to average_age, and we can come down here to HAVING and say HAVING the average_age greater than 40, and when we run this it works perfectly, and you'll notice that the name of the column was actually changed. Now this AS isn't actually 100% needed; it's kind of implied, even if we get rid of it, it's implied there's like this AS in there somewhere, um, but we don't have to have it. If we took it out and ran it like this, it would still work exactly the same. So that is how we can use LIMIT and aliasing in SQL. And congratulations, this is the end of the beginner series in my SQL. In the intermediate series, we're going to take a look at things like joins, unions, case statements, subqueries, and window functions.

Hello everybody. In this lesson, we're going to be taking a look at joins. Joins allow you to combine two tables or more together if they have a common column. That doesn't mean the column name has to be the exact same, but at least the data within it are similar that you can use. There are several joins that we're going to look at today, like inner joins, outer joins, and self joins. These are the two tables that we'll be working with the most throughout this lesson: we have the employee_demographics table as well as the employee_salary table. Now within the employee_demographics table we do have this employee_ID column, and if we look at the employee_salary, we also have the employee_ID column. So in this instance the column name is actually the exact same, and of course the data inside of it is also very similar. So let's start by writing out an inner join. This is probably one of the most common joins, one of the most simple joins as well. An inner join is going to return rows that are the same in both columns from both tables. So let's see how we can actually write out this join. Let's come right down here and let's copy this. This will be the first table that we start with, and we'll join this salary table onto this demographics table. So what we need to do is we need to come right here and we need to say JOIN. Now by default JOIN represents an inner join, although we can write INNER JOIN here to make it more explicit. Explicitly writing out INNER JOIN, then we're going to come up here and we're going to say employee_salary. So we're selecting everything from the employee_demographics; we're doing an inner join on the employee_salary. Now we have to tell MySQL exactly what columns we're supposed to be joining on. So I'm going to hit enter and I'm going to hit tab. Now you don't have to hit tab; it just looks more code-like, and it's easier to read, and that's how I've done it for other programming languages as well. So that's how I'm going to show you how to do it. What we need to do is say ON. Now this keyword is going to allow us to say we're joining the demographics table to the salary table based on these two columns. So from the demographics table we're doing the employee_ID is equal to, and then in the salary table it's also the employee_ID. Let's do employee, and you spell that right: employee_ID. Now if we try to run this, and let's do this, we're going to get an error, and let's bring this up. It's going to say column employee_ID on the IN clause is ambiguous. Now what does it mean ambiguous? That means that it doesn't know what table this employee_ID is from; is it from the employee_demographics table? Is it from the employee_salary table? We don't know because it's ambiguous. Now what we can do is we can specify it by saying employee_demographics.employee_ID and then employee_salary.employee_ID. Now if we run this, we're going to get the output that we're looking for. And let's take a look at this real quick. Let me bring this up. So we're pulling everything from the employee_demographics; that's right here all the way through the birth_date. Then we're pulling the employee_salary table; that's the employee_ID all the way, let's scroll over, through the department_ID. So we're basically pulling in all of the rows or all the columns from both tables, but we're not pulling in all of the rows. Remember, an inner join is only going to bring over the rows that have the same values in both columns that we're tying on. So in this employee_ID, we're missing number two. Are we missing any other ones? No, we're only missing number two. Let's go back up, and I'm going to run both of these tables, and we're going to take a look. So let's run this. So you'll notice in the employees_salary table we have a number two right here, and that's Ron Swanson, but in the employee_demographics table we don't have that. I believe that Ron Swanson did this; that Leslie Knope would not know when his birthday was because he didn't want to bring that information. I think that makes the most sense, although Ron was not willing to give a comment on that. Now if we run this again, you'll notice that two is not in there. Since two is not in the employee_demographics table, the employee_ID 2 is not going to be populated or brought over into this output from the employee_salary table.

Now really quickly, this is honestly, uh, giving me some anxiety because this is so incredibly long. Something that I mentioned in the beginner series is that you can use something called aliasing when using joins, and it's really helpful. This is what I mean. So right here we have employee_demographics; we're going to call this DM. You can also do AS DM; you can do AS SAL; these are just short names for demographics and short name for salary, and we can replace these and say dm.employee_ID and s.employee_ID. Oh, that looks so much better. Now we're going to run this, and it'll be the exact same output, but now we're using these aliases, which just makes it so much easier to read.

Now one last thing that I want to show you while we're just looking at the inner join is selecting the actual columns. Let's say we wanted to select the employee_ID and we wanted to select age and then we wanted to select their occupation. If we try to run this, we're going to get an error, and it's going to be almost the exact same error that we got before, which is column employee_ID in field list is ambiguous. So in our field list, which is right up here in the SELECT statement, we have this employee_ID; it does not know which employee_ID to pull from, whether it's the demographics or the salary. So we have to tell it which one to pull from. So let's pull it from the demographics by saying dm.employee_ID. Now when we run this, we're able to get information from both tables in our output without having all of the information. And if there are columns that are similar in both tables, we have to denote that by using this alias or the table name. All right, so that is inner joins. Now let's move down here; I'm going to copy this, and we're going to come right down here, and we're going to look at outer joins next. And let's put that right here. Now for outer joins, we have a left join and we have a right join, or a left outer and a right outer join. A left join is going to take everything from the left table, even if there's no match in the join, and then it will only return the matches from the right table. The exact opposite is true for a right join. Let's see how this actually works. Let's start by changing this to a left join or a left outer join; they're both the same, and you can use

them uh, similarly, I'm just going to say left join, and we're joining it on the exact same things, and I'm going to take everything, so I think that'll be easier to visualize, and I'm going to run this. Now you may notice that this looks exactly the same, and that's for a very good reason. It's because in the left table, which is the employee demographics table, the from statement that's our left table, and then the join, what we're actually joining on, that's our right table. So this is our right table. So since we're doing a left, it's taking everything from the employee demographics. Now remember, the employee demographics didn't have Ron Swanson; it had no information, so everything in the right table had a match. Let's change this to a right join, and what this is going to do, and I want to make it all cap, make it all the same, what this is going to do is it's going to take everything from the employee salary table, but if there is not a match in the employee demographics, it just will have nulls. Let's go ahead and run this. So now it looks a little bit different. Now we're taking everything from the employee salary, so we're taking Ron Swanson, but if there is not a match, it will still populate that row, but it'll have all nulls in it. Then any of the information that is overlapping are the same, it will bring over, so employee ID is matched to employee ID 1, then we'll bring all that information over from the left table, and that's essentially what a left and a right join is. With a left table, you're taking everything from the left table and then matches from the right table. If you do a right join, you're taking everything from the right table, but only matches on the left table, and again, it populates it with nulls.

Now let's go down and look at our last type of join that we're going to look at, and this is a self, let me spell that right, a self join. Now what is a self join? It is a join where you tie the table to itself. Now why would you want to do this? Let's take a look at a very serious use case. Let's do select everything; let's do this from employee salary, and let's run this. Now let's say it's December 1st, and the employee and Rex Department decided to do a secret Santa, and they wanted to assign based off of their employee ID the person who they're going to have as a secret Santa. We can help orchestrate this very easily using MySQL; well, very easily is subjective, I guess, but let's take a look at how we can do this. So just like any other join, the first thing we're going to do is select everything from employees salary and then say join, and then we're going to say employee salary again, so we're tying it to itself. Now when we come down here, and let me do this, when we come down here and we say on, we have to specify which table we're pulling from; are we pulling from the left table, which is like the first table we're pulling from, or we pulling from when we're joining on the right table? We need to be able to distinguish these two tables because they are the same, so I'm going to say emp1, and I'm going to say emp2, just to say this is employee table one and employee table two. So we're going to tie them based off the employee ID because we know those will be the exact same because we're pulling from the same table, so we'll do emp1.employee, and just so you know, if it populates like this, you can hit Tab, and it'll auto finish that for you, and emp2.employee ID. Now if we run this, let's do this, the output that we're going to get is literally just a one for one match; it's all the columns and all the rows because they all match exactly. But now what we're going to do is we're going to assign an employee ID to the next employee ID, and that will be their Secret Santa. So just keep it really simple; the next highest person with an employee ID that is their Secret Santa. So let's do an employee ID plus one is equal to employee2.employee ID, so we're adding one over here, and we're saying that's equal to this employee ID over here. Let's run this. So now you can see Leslie Nope is now going to be assigned to Ron Swanson, who has an ID of two. Ron Swanson is going to be assigned to Tom Haverford, which I'm sure he's really happy about, and so on and so forth.

Now let's bring this down here, and what we're going to do is try to simplify this and simplify this output a little bit, because this is a little bit chaotic down here. So we're going to specify what columns we want in our output. What we're going to want is the employee ID, first name, last name, and then employee ID, first name, last name of the person who they got for Secret Santa. So we're going to start with emp1.employee ID, and we can call this, we'll just say as emp_Santa. Then we'll do a comma, and we'll come down. Now I need to spell employee right, so we have our employee ID, and now we need our first name and last name. Now remember, Leslie Nope is going to be the secret Santa for Ron Swanson; I don't know if I made that clear, but that's I guess how it works. So now we need to do uh emp1. and we'll do first_name, and we'll do as, we do as first_name_Santa. We can do a comma; we'll do the exact same thing except for the last name, so last_name as last_name_Santa. Now all we have to do is do a copy all this, bring it down here, and change this to two; we're pulling from the second table, and it'll look just like this, and get rid of this comma, and this is done. Let's run it, and let's bring this up, so we have employee_Santa, first_name_Santa, Leslie, last_name_Santa, Nope, then employee_Santa, and we actually need to change these names; that is one thing we need to do. We'll just change it to employee_name, first_name_employee, and last_name_employee, and now when we run this, we have our Santa, and then we just have the employee who this person is going to be the Santa for. Now this is kind of a silly way to look at it, but in essence, this is exactly how a self join works.

Now the very, very, very last thing, I promise you, last thing that I want to show you is how we can join multiple tables together. So we're going to say joining multiple, you spell that right, multiple tables together. Now not just one table together to another table; I'm talking about one table to another table to another table. So let's go all the way back up; we're going to take this right here and bring it all the way down, and what we're now going to do is we're going to tie in this table right here, the parks department. Let's actually look at this table, and let's select everything real quick, and let's run this. Now let's go down here, and we're going to say select everything; we'll do this from Park_departments, and let's run this. Now this is something called a reference table; this is not a table that most likely you'll ever add a bunch of information to; it's there to reference that we have these department names. Tables like the salary table or employee demographics table are going to change pretty often as people get raises or as they get older with their age; those are going to be updated fairly often, whereas this parks department table is just there for reference. Now if we look down here in the columns, we have a department ID, then we have a department name, so we have the ID and the name of that ID. If we run our join and we scroll all the way to the right, you'll notice we have a d_ID; this stands for Department ID; that's in the salary table. So what we want to do is join this department ID to the department ID from the parks and rec. So what we can do is we're going to say inner join, and now we're going to join, let's scroll down just a hair, there we go. Now we're going to take this and do it off this, so we're going to call this PD for short, and we're going to say we're joining it on. Now we cannot join this parks department to the employee demographics table; why is that? Well, the employee demographics table only has employee ID all the way through birth date; there's no common column that we can tie to this parks department. The only table that has a common column is this department ID in the salary table, so what we need to do is actually take s, so we'll say Sal., and then we're going to say d_, and we'll say Department ID is equal to the PD., and we need to take the department ID. Now notice these are not the exact same name; they are a little bit different, but they have the same values. One thing I forgot to mention is that in this parks department, there's no repeating; that's why it's a reference, whereas in the salary, the department ID repeats several times cuz multiple people are in the same department. So this reference table also usually does not have duplicates, uh, just one other thing to note, but we have now tied it successfully. Let's try to run this, and if we come down here, go all the way to the right, we now have the department ID 1111111 and the department ID and Department name, Parks and Recreation, Healthcare, Public Works, Finance, Public Works, and Parks and Recreation. So this worked perfectly. So this is how you can tie multiple tables together if you have common columns between them, even though employee demographics has no column that's related to the parks department table, we can still tie them together based through this employee salary because employee demographics can tie to employee salary, employee salary can tie to the parks department, and that really is the majority of what you need to know in order to use joins. Well, now in the next lesson, we're going to be taking a look at something called a union.

[Music]

hello everybody, in this lesson we're going to be taking a look at unions in MySQL. A union allows you to combine rows together, not like columns like we were doing before with joins where one column is next to the other; a union allows you to combine the rows of data from separate tables or from the same table; it's up to you, but you do that by taking one select statement and using a union to combine it with another select statement. Let's see how this actually looks. So what we're going to do is right after this select statement, we're going to come here and say Union, then we're going to go right below the union, and we're going to do another select statement. So we're going to copy this, place it right here; instead of the demographics table, just for example, we'll do the salary table. Now if we look at the demographics table, let's say we want to take age and gender, and let's go and take a look really quickly at the salary table, and let's say we want to take a first name and last name, so we'll do first_name and last_name. Now let's go ahead and run this and see what it looks like, and let's pull this up. So as you can see, we have age and gender that's from the very first select statement, and that's also the column names, but then we have all of the data for the age and gender, and then below, if we move this over a little bit, we have the last name and first name from the employee salary table; it's just down here. Now what I just demonstrate is that this doesn't always work for everything; you can't just combine random data together because this is bad data; we shouldn't have age and gender mixed with first name and last name. Really, when you're using this, you need to keep the data the same. So for us, we should take the first, I'll actually just copy this, the first and last name from the employee demographics as well, and let's run this, and now we have all the names from all of the tables. Now you may be thinking, where did all the other data go? Before we had a lot of rows, but now we only have a unique row for each one. Well, by default, this is actually a union distinct, and if you remember, distinct is only going to take unique values. So when we're doing this, Union is going to remove all the duplicates, and the first name and last name from salary overlaps a lot with the employee demographics table, so when we ran this, the only one that's actually somewhat unique to one table is that in the employee salary table we have Ron Swanson, whereas we don't have that in the employee demographics. Now if we wanted to show all of them without the distinct, there is something called a union all. If we run this, now we're going to get all of the results without removing any of the duplicates. So if we scroll down, we're going to have duplicates in here, but we're just showing all of the results from this table and from this table.

Now that we know how to actually use a union, let's look at a use case. So let's go right down here, and let's copy this, why not, and let's put it right down here. Now let's say in the employee demographics we wanted to take the first name and last name where the age is greater than 50, and let's run this; there's only one person, but let's label them; let's add a label; we're going to say comma old, so this person is old, and if we run this, it says first name, last name, and old, and we can even call this as label, and if we run this, the label is old, so Jerry Gergich is the only old person in this demographics table. Now why are we doing this? Well, the parks department is trying to cut their budget a little bit; they want to identify older employees that they they can push out, and they also want to identify high-paid employees who they can reduce their pay or push them out to save money. So we just identified someone who's older who we're going to want to try to push out, but let's in the same output find people who are also highly paid. So now we can come down here; we can say Union, and let's do this like this; I need to spell this right, all right, Union, and let's take this; we're not going to be using this exact same query, but we actually need to from the salary table, so the employee salary, so we also want the first name and last name, but let's say where there salary is greater than let's say 70,000; that's a lot of money; if you're making more than 70, uh, for sure the parks department is going to try to get rid of you, but for the label, we're going to change it to a highly paid employee. Now let's go ahead and run this. So now we have Leslie Nope and Chris Traeger; they're both labeled as highly paid employees. Now 50 I think is just a little too low, um, if I'm being completely honest; I think we need to change this, and we should do a union and then add another select statement. Let's bring this down; I think the 50 is too low; let's change it to 40, and let's add one more thing; let's say and the gender is equal to male, and then we'll go down here and say where the gender is equal to female, cuz we want to separate this out, so we want to know who's the old man, oh, that's actually old lady, this is the female one, and for up here where it's male, we'll say old man. So we have three different select statements using two separate unions; we're selecting the first name and the last name in all of them, keeping the data consistent, and then in our third column, we're labeling it either old man, old lady, or highly paid employee. Let's go ahead and run this, and let's look at our output. Now you may notice something really quickly that Chris Trager and Leslie Nope are an old man and an old lady, and Leslie Nope and Chris Traeger are both highly paid employees, so these people meet multiple criteria. So let's actually order by, and then we'll do first_name, comma last_name, because we want to order by these to see, so let's run, and now we can easily see that Chris Traeger is both of these, Donna is just an old lady, Jerry is just an old man, and Leslie is both an old lady and a highly paid employee. So now we can send this to whoever we need to send it to to make sure that these people get looked at first so that our job is still secure; the job market is tough these days; you got to do what you got to do. So that is how we use Union, and let's just take one more look at it; there we go. So, so this is how we can use unions; it's kind of a real use case; I've done something very similar to this in my real job, but you know, this is just an example of how you can have multiple select statements all combined or combining the rows using a union. In the next lesson, we're going to be taking a look at case statements.

[Music]

hello everybody, in this lesson we're going to be taking a look at string functions. Now string functions are built-in functions within MySQL that will help us use strings and work with strings differently. Now we're going to look at a ton of different ones; they all have different use cases, but I'll try to walk through some of those as we go along, but we'll look at a lot of different string functions in this lesson. We'll start off with one that's really simple; this one is called length. So if we select, then we say length, and let's say we put, put in, and I don't know why it's popping up like that, let's say we put in something like sky, or Skyfall, or really anything, if we run this, it's going to give us the length of how long this string is. So if I come down here and we say select everything from employee_demographics, and let's add a semicolon here, let's run this one right down. Now what we can do is we can look at how long each person's name is; what we can do is just take the first name, but then we'll also do the length of the first_name. So if we run this, now we get Leslie, Tom, Jerry, Donna, and it gives us the length of their name. If we wanted to, we could even order by this, so we could do order by, and we could just do two for now, and we can order by the length from the shortest name all the way to the longest name. Now one use case that I've used length for in my actual job was when I was working with phone numbers; I wanted to make sure that they were exactly 10 characters long, otherwise something went wrong somewhere in the data cleaning process, so I would go and look at the length, and I would make sure they're all 10; if anywhere above 10, I will go and specifically look at those and try to clean those and fix those up.

Now let's go on to the next one, and these next ones are pretty simple ones, at least I think they're fairly simple; we're going to look at upper first, and it's doing the same thing as the other one; we'll do upper, and let's say we're going to do Sky. If we select upper Sky, it's going to give us an all uppercase, or we can copy this, and we can do lower. So now let me add semicolons, otherwise it's going to drive me crazy, uh, let's try this lower. Now it's going to do all lower, even if I make it all capital. So if I say all capital Sky, it's going to make it all lower. So if we come back up, let's copy this, and instead of doing the length, now we'll do upper; let's go ahead and select this, so we have Leslie, and then we have the upper first name, so upper all case Leslie. Now this is actually really good; this is really helpful, especially with standardization is what I found a great use case for it, because sometimes it'll be all capital Tom, and sometimes they'll put it in as T lowercase om, uh, and just making them all uppercase or all lowercase can help correct those really simple standardization issues within a single column. The next one that we're going to look at is trim. Now there's multiple trims; we have trim, left trim, and right trim. Trim is basically going to take the white space on the front or the end.

and get rid of it, which is really really helpful. So what we're going to do is we're going to come right here, say select, and we'll start off with trim; and let me add a semicolon every time; so then we'll do trim; and for our actual string, we'll do something a little bit odd; we'll do some spaces; and then we'll do a sky; and then we'll add some spaces. Now let's run this; and add our semicolon; that's going to be the end of me in this lesson, just adding semicolons. Now it fixes it completely.

Now what if we don't add sky at all? We'll just keep it like this. Well, you can see that there's spaces before and there's spaces after, but that's what trim does; trim gets rid of the leading and the trailing white spaces. Now if we come up here and we just do the left trim, it's only going to remove from the left-hand side, so we're only getting rid of the left-hand side white spaces; this right-hand side, as you can see, is really long; it's still there. And if we do our trim, we go ahead and run this one, it gets rid of the white space on this side, but it doesn't get rid of the left space on this side.

Now let's keep going; we have a lot to cover still. We're going to move on to what I think is probably my most favorite string function, if I'm allowed to have a favorite string function, and that's substring; but I'm going to kind of work us into substring a little bit by looking at two smaller functions, which is left and right. So let's select everything; and we'll do that from the employee demographics again; it's add our semicolon. Now I'm going to run this. Now I want to get the first name, and I'm going to do left of the first underscore name just like this. Now when you're using this, is actually going to be an error. Let's see if I highlight over this if it'll tell me what the error is; it says the parentheses is not a valid position; they're expecting something else; and basically what they're telling us is that this is not how it should be written; we're looking for a different value; that different value is actually a number; we're going to do comma, and let's do four; that's what it was looking for; it didn't want this at the end; it needed this comma four; and what we're actually specifying is how many characters from the left-hand side do we want to select; so we're selecting the first name, and we're going from the left four characters. Let's go ahead and run this; and so we have Leslie, Tom, April, all the way down; you can see that there's only four characters in each one; so someone like Chris, the S is no longer going to be there because we're only looking at the first four characters.

Now we can do the exact same thing, and let's actually copy this uh down here; so we'll come, we'll go like this, make this a little more professional, and we'll do right; so now we'll do right; if we do the right four, it's going to go from the right-hand side of the string and go left four; so we're looking at the far four most right characters. Now this can be useful in certain instances, but if I'm being honest, I don't use these that much; for the most part, I'm pretty addicted to using substrings; I love substrings; I think they're fantastic; and let's look at substrings like this; so a substring is going to allow us to do a few different things; let's do first_name; the second thing that we put within this function is the position that we want to start at; so let's say we want to start at the third position; and then we specify how many characters we want to go; so with this we specified four, but let's just do two; so now we're going to the third position, and we're going over to the right two characters. Let's go ahead and run this; so with Leslie, we get SL; so we go one, two, and three; we start, start at the third position, and then we take two characters, the S and the L. I have found this one to be extremely extremely useful. Let's take this for example; let's do comma; let's do birth_date; and let's run this; I'm keeping everything in here, although it might be a bit much; let's say we have this birth date, and this middle column is the month, and we're running some, you know, query; we want to find the month that everyone is born; so we can do that very easily using substring, and we wouldn't have been able to do this very easily using left or right. So now we're going to take this birth date, and we're going to use this substring, and we want to select these middle characters; so what we need to do, since it's all standardized, we do 1, 2, 3, 4, 5, 6; we start at position six, and we want to select one and two. Let's go ahead and run this; and now we've pulled out all of the months; so we can say as birth_month; month; and now we could save that, put it into a temp table, add it as a new column in our table, whatever we want to do, uh, but now we have this information that we desperately desperately wanted to know. So that is left, right, and substring; again, substring is, it's fantastic. Now let's keep going; the next thing that we're going to take a look at is replace. Now replace will replace specific characters with a different character that you want; so let's actually copy all this right here because I don't want to keep writing this out; and, and we'll say select everything. Now what we're going to do is we're going to take the first_name, and then we're going to say replace; and then we'll also do the first_name; but we can specify what we want to replace and then what we want to replace it with; so we have two more parameters that we need to put in this function; so let's say a, and let's replace it with a z; let's just see what that does; let's go ahead and run this; and so now when we see the letter a, and we are specifying a lowercase a, like Mark, that is replaced with a z; so that's really all replace does; specifies what you want to replace and then what you're going to replace it with.

Now let's take a look at the next one, and we're going to take a look at a function called locate; so if I say select, and let's do locate; I'm going to give it a string; I'll say Alexander; that's my name; and I'm going to specify what I'm looking for; so let's close this parenthesis; the string that we're actually looking for comes first; so what we're going to do is I'm looking for the letter X in my name; so we'll do X and Alexander; let's go ahead and run this; and it tells us that it is in position four; so we have 1, 2, 3, and four; that's where our position is; that's where it locates that sequence that we're looking for. Now if we pull this down here, place this right here, and we'll change this locate; now let's say we're still looking at the first name, but we want to locate people that, that have an an like this in their name; let's go ahead and run this; and we get zeros for everybody except for an and Andy; so this might be something where we put it into a CTE or a temp table; then we can filter down based off of these results to where it only equals one.

Now the last one that we're going to take a look at, and let's go right here, we're going to do first_name, last_name; now this one is super super useful because what we can do is have a concatenation of multiple columns; so let's go down right here; so we have first name and last name; but if we come down and we say concat, we can then combine these columns into one single column; so we'll do concat; and we'll do first_name; and then comma; last_name; and if we run this, it's going to be Leslie and nope combined into Leslie nope. Now this doesn't look perfect, right? We don't want it to look like that; all we have to do is come in here, and we could do a little space; so we'll add of space in there; and if we run that, now we have Leslie and nope; and we can call this as full_name; and this is something that I've done a million times in my real job where there's multiple columns; we want to create one column out of it, or take two columns and create one column; happens all the time; so this concat is really really helpful to combine those columns really quickly. So that is all we're going to take a look at in this string functions lesson; in my full course, you can find in the description below; I also have lessons on numeric functions, date and time functions, converting different data types, all in the functions [Music] module.

Hello everybody; in this lesson, we're going to be taking a look at case statements in MySQL. A case statement allows you to add logic in your select statement, sort of like an if else statement in almost all other programming languages, or even things like Excel. Let's see how this actually works; so let's bring this down, and let's take this employee demographics table, and let's take the first name and let's take the last name, and let's add a case statement; how we need to do this is we have to say case; so that's going to signify that we're starting a case statement; and then I'm going to go over here and say tab; so this is where our logic comes into play; so I'm going to say when the age, let's say, is less than or equal to 30, then; so I'm saying if the age is less than or equal to 30, then what's going to happen; we'll just keep it really simple for now; we'll just say that this person is Young; and then if we want to end the case statement, we'll come down here and say end; so this is a complete case statement. Let's go ahead and run it; and let's take a look at the output; so we have the first name, we have the last name, and then we have this case statement right here; and if their age is less than or equal to 30, they're young. Let's actually add the age right here just so we can visually see that as well; so we have the age; so this person is the only person who's under or equal to the age of 30; that's April; so she has a label of young. The great thing about case statements is you can add multiple when statements; so we can come down here and say when; and then we could do something like when age; and maybe we'll say between; so I don't know if in previous lessons we've looked at between, but between just says between this number and this number; so we'll say between 31 and 50; if they're between 31 and 50, well, good night; uh, that person is old; so we're going to have it just like this; we're going to run it; and now we have a lot of people who are old; these are all people between the ages of 31 and 50; but we still have more people outside of the age of 50 or older than 50; so we could do when the age; and now we can say greater than or equal to 50; and we're going to say then; and then we're going to say on death's door; uh, because good night; if you're over 50; my parents are going to love me for this one; so let's go ahead and run this; and then if we look at this, we have on death's door right there. Now this is huge; this is massive; so let's actually name this, and we'll just say as at the end of end; so right after end, we'll say as age_bracket; and let's run this; and this looks a lot better; so now we have this age bracket just signifying kind of where people are at; and most people are quite old, or Jerry, uh, you know, can't catch a break that guy.

Now let's go down and let's take a look at a different table; so let's select everything; we'll do from employee_salary. Now that we have our employee salary table, here is the scenario that we are given: The pony Council sent out a memo of their bonus and pay increase for end of year, and we need to follow it and determine people's end-of-year salary, or the salary going into the new year, and if they got a Bonus, how much was it. So the first thing we need to do is we need to get the pay increase and bonus; and their pay increases look like this: So if they made less than 50,000, then that equals a 5% raise; very generous; and if they made greater than 50,000, that equals a 7% raise; very very generous; lastly, if they work in the finance department, that equals a 10% bonus; just cash that goes into their bank account; very very generous; but only the finance department gets it. So these are the guidelines that the pony Council sent out, and it is our job to determine and figure out those pay increases as well as the bonuses. So let's come right down here; we're going to have our salary; employee; I actually want to be able to see these; let me pull this up just a touch; there we go; so we want to be able to write this out; first thing we should do is just select the columns that we need: first name, last name, uh, probably salary as well; and now what we can do is determine this first one, which is if they make less than 50,000, they get a 5% raise; so let's say case; and I'll also add end in here; and we're going to say when their salary is less than 50,000; what's going to happen; then we say then salary; so we're taking their initial salary, and we're saying plus; and we're going to do salary times 0.05; and if we run this, should work; let's pull this up really quickly; so April Ludgate, she made under 50,000; so she got a raise; and her new salary is 26,250; we can actually call that; we'll say uh as p_ actually, let's do new_salary; that's their new salary; and let's run this; so the new salary is 26,250; Andy Dwyer is now making 21,000. Now this calculation, you can do it different ways; we can do exactly like this, or we could just do times 1.05; should be the exact same thing, uh, just however you like to write it out; it's just, you know, adding it or multiplying it by this; so let's take this; and now we're going to say when it is greater than 50,000; so let's say greater than 50,000; they get a 1.07; so this is the 7% increase; this is a 5% increase; this is a 7% increase; and let's run this; and let's put this up here; so now if they made greater; so 50,000; that's 75,000; they got a 7% increase. Now unfortunately, we did not make the rules; the pony council did; and the people who made exactly 50,000 unfortunately were not part of those brackets, uh, and that just wasn't up to us; we couldn't control that; so unfortunately, Tom Haverford and Jerry Gergich just didn't get raised this year; and that's not our fault; okay, that's not our fault.

Now the next thing that we need to do is determine the bonuses. Now let's come right back up here really quickly, and let's just copy this because what we need to determine is how we know that somebody is in the finance department; because if they're in the finance department, that means they get a 10% bonus; that's really important. Now it's not in the employee demographic; don't have anything about the department; but if we look in the salary, and we run this, we do have the department ID. Now let's open up and let's pull this up right here; we'll look at the park department; and in the parks department, here we go; the finance is Department ID of six; so if we're looking at the salary, there's only one person who's in uh Department ID equal to six; so what we can do is another case statement; we can say comma; we'll do case; and end; and we'll do another one; we're going to say when D_ID; so when the department ID is equal to six; then we're going to give them a bonus; so we're going to say salary times 1.10; and we'll call this as bonus; let's go ahead and run this; and let's pull it up; so he gets a $7,000 bonus this year; that's Ben Wyatt, uh, because he was part of the finance department that just did an exceptional job this year, apparently, according to the pony Council. So that is how case statements work; they're really powerful, really useful; I honestly use them quite often; and they're just a way to really add some logic and some, you know, labeling, or even do calculations like we did right here with the salary. In the next lesson, we're going to be taking a look at subqueries in my [Music] SQL.

Hello everybody; in this lesson, we're going to be taking a look at subqueries in MySQL. Now a subquery is basically just a query within another query; we can do this in a few different ways, and I'm going to try to show you a lot of the different variations within this lesson. The first way that we're going to use a subquery is in the where clause; then we'll take a look at the select and the from clause also. Let's take this demographics table that we have down here; what if we only wanted to select the employees who worked in the actual parks and rec department? Well, we could do that if we had a few joins; we have the salary table, and one actually represents that they work for the parks and rec; if we come over here and we open this up, we can see that parks and rec is the department ID of one; so we do have that option; we could just join these two tables together, but sometimes we don't want to do that, and we'll use a subquery. Let's see how it works in the where clause; so let's go ahead and get rid of this; so what we're going to do is we're going to say select everything from employee demographics where; and now we want to pull, because this is the salary table, we want to pull employee IDs where the department ID is equal to one; but remember we're querying off of this table; so let's actually pull this up; this is what we're working with; so we want to say where the employee_ID, that's referencing this column in the demographics table, is in; what we're going to do is we're going to do a parenthesis here; and we can even come down and put a parenthesis down here; so what we're going to do now is write our query, which is our subquery; and this is our outer query; so now we're going to write an entirely other query within this; we'll say select; and now we're going to say employeeid; and let's just bring this over; I usually have it something like this; and I'm going to try to bring this down a little bit; so select everything; and then we'll do from; and then instead of employee demographics, we'll do employee salary; and let's just format this a little better; so select the employee ID from employee salary; and remember we wanted to do where the department ID is equal to one. Now let's bring this back up; and this is what the query is going to look like. Now just by itself, let's run this subquery, or this inner query; when we run this, it's going to create this list of just employee IDs where the department ID is equal to one; so when we say where the employee ID from the employee demographics table is in, it's going to try to match those employee IDs to this list of employee IDs; so just remember 1, 2, 3, 4, 5, 6, and 12. Let's go ahead and run this entire query; now we have 1, 3, 4, 5, 6, 12; if you remember from previous lessons, the two is Ron Swanson, and he's only in the salary table; so since we're doing just the employee demographics table, he's not in here; so what we're doing is we're selecting everything from the employee demographics where the employee ID in this table matches or is in the select employee ID from the salary table where the department ID is equal to one; in essence, this is what a subquery is; it's a query within a query. Now what would happen if we have the employee ID, but we also wanted to say the department ID, because we just wanted to view this? Let's go ahead and try to run this; we are going to get no output, and we're going to get an error that says operand should contain one column; the operand referring to this entire thing right here, cuz this is an operator; so

This is our operand, and we're returning two columns in here, which is saying we cannot do; we have to only have one. So now, if we run this, it works perfectly well. And let's bring that down. Now we can also use the subquery in a select statement, so let's take a look at that next. Let's go down here, and let's say we want to do select everything from employee salary, and let me spell that right. Let's say we want to look at all the salaries just like how we have it now, but in a column next to it, we also want to compare it to the average salary for everyone, so we'll be able to see, you know, whether somebody's salary is above average or below average. So what we would try to do potentially is do something like, uh, first_name, salary, and average salary, and we try to run this, and of course, we're going to get an error; it's going to basically tell us that we need to group by if we're doing this.

Let's go back down, and let's actually add that group by, and we'll say group by first_name and salary, and we'll look at this output, and this is not looking good at all; it's just looking at the average salary for each unique row, which is Leslie, 75,000, so the average is 75,000. This is not what we're looking for; this is not what we want. Here's what we really do want: we want to just take the average salary of this entire column regardless of group by or anything else. So let's get rid of this and let's see how we can do that. So let's come right down here; we're going to say select, select the average salary, and then we're going to say from, at our parenthesis because this is our subquery, from the employee salary table, just like that. Now, if we run this, we should get the exact output we're looking for, so the average salary is 57,200, and we have our salary right here, so we can compare really quickly just like that.

And we can also use a subquery in the from statement. So let's go down here, and let's say select everything from employee_uncore_demographics; let's have it autocomplete for me, so we have the employee_demographics table. Now let's create a group by based off the gender column, that add some aggregated functions, and I'm going to show you how you can use this as a subquery. Let's go up here, say gender, and then we'll just go ahead and add our group by, so we say Group by gender as well. Now let's add a few things; we'll do average, we'll do average age, and then we can do, let's just do all of them based off the age; we'll just do age, min of age, and count of age. Woo! When I try to write fast, it doesn't always go right. So we have this; let's run this, and this is what our output is going to look like.

Now, what if we wanted to get the average of the oldest age or the average of the smallest ages or, you know, see what the average count is for males and females? Well, we can't do that given this table, but let's do something right here: select everything, and we're going to say from, and in our from statement we're going to have a parenthesis; we're going to paste our select statement and then close the parentheses, so we're going to select everything from this output that is right down here. So if we run just this, we're going to get an error, and forgot this was going to happen, but every Drive table must have its own alias, so you have to name a table; I forgot it does that. All we have to do to fix this is just name it, so we'll say as, and we'll say aggregated_table; we'll just call it aggregated_table. So let's run this, and we get the exact same output, but here's the neat thing: is we can now select; we can do gender, and these are actually the column names now, so I can do the average of this column right here, but I can't do it just like this because it's going to give us an error, and I'll show you why in just a second: says unknown column age in field list. So what it's saying is is we're trying to perform an aggregated function on the aggregation of an age column, but we don't have an age column in our table right here. Let me run this again; we have a column named this exact thing, so we actually need to do is do this backtick and backtick; this is the actual name of the column; it's not an aggregation anymore. The backtick on my laptop is right above the tab on the far left, left-hand side, um, right under the Escape; that's where mine is, uh, so these backticks, it's not a quote like this; it's a backtick, so you just need to find that on your keyboard. But now, if we run this, it looks like we encountered another error: says in aggregated query without Group by; that's right; now we need a group by, so now we need a group by gender. Sometimes you got to figure this out on the fly, and it should work; there we go. So now we can perform aggregation on this table. Now this doesn't actually work perfect because we're still grouping by the female male, but let's get rid of this for a second, and we'll get rid of this group by entirely, and if we run this, we're now looking at the averages of this column right here: max age. Now, when you're doing something like this, it's actually really smart to rename these; we'll say as average_age; we'll say as max_age, and it makes it so much easier; you won't have to do these backticks anymore, um, and as min_age, and so on and so forth, and I would probably format this better and stuff like that; we don't have to go through everything right; I'm just kind of giving you an example, but then when we're using this table, these columns are actually named this, so I don't have to do these backticks anymore; I can just take this whole thing, oops, get rid of that backtick; now I can just take this column cuz this is the column name. So let's go ahead and run this, and it's still going to work perfectly. So this one's pretty cool because you're basically creating this kind of like a temp table, um, you're just creating your own little output, then you can query off of it, and you can do, you know, more advanced calculations this way; it's actually really useful, but there are better ways to do something like this, uh, like a CTE or a temp table that we'll look at in the advanced series, but this is at least how you can do it, and you can actually try it out using subqueries. So that is all we're going to look at today for subqueries. In the next lesson, we're going to take a look at window functions.

[Music]

Hello everybody. In this lesson, we're going to be taking a look at window functions. Now, window functions are really powerful and are somewhat like a group by, except they don't roll everything up into one row when grouping. Window functions allow us to look at a partition or a group, but they each keep their own unique rows in the output. We're also going to look at things like row numbers, Rank, and dense rank at the end of this lesson. So before we jump into writing a window function and seeing how the syntax works, let's actually write out a group by, and then we'll compare the two when we actually do write the window function. Let's say we want to take this demographics table and we want to take this gender and compare it to the actual salaries. So what we actually need to do: we need to say join, and we're going to join on the employee salary; let's go like this, get rid of all of this, and we'll do salary, and we're going to say on, and let's do dem and sal for the aliases; we'll see dem.employee_ID is equal to sal.employee_ID. Now we're going to come up here, and we're going to say gender, comma, and we want to look at the average salary, and we need to get rid of this right here, and we need to come down to the bottom and say Group by gender. Now let's go ahead and run this query, see if it works, and it did. So we have our gender, and we have our average salary from our salary table, and we can rename this as average, and we'll do average_salary just like that. So this is how Group by works; it rolls everything up into one row. Now let's try doing something pretty similar, except we're going to use a window function. Let's come right down here and let's paste this, and let's start writing out our window function. Now we don't have to use the group by; we're going to go ahead and get rid of that, and right here for gender we can keep that the exact same; all we're really going to change is this part right here; we're going to say average salary, and that is part of creating a window function; typically with a straightforward window function, all we have to put is over with a closed parenthesis; this is going to say we're looking at the average salary over, and normally in here you'll specify something, and we'll get to that in a little bit, but we're just going to look at an average salary over everything. So let's go ahead and run this output. So this is going to look a little bit different right; so the male and female all have their own individual rows, which is not the same as group by, and this average salary is looking at the average salary of everybody; we're not breaking it out by the gender like we did up here; here we rolled it up; now we're looking at the average salary for the entire column. Now what we can do is actually partition by. Now Partition by is going to separate it out kind of like grouping it, so let's say partition, Partition by, and we'll say gender. So just like when we did the group by, the group by rolled everything up into one row; this is not going to roll everything up, but it is going to perform this calculation based off of the different genders, the unique values in this column. Let's go ahead and run this, and if you'll notice the female is 53,723. Now let's go compare these; I'm going to run this and this query; let's run this. So if we look at our group by, it's the exact same numbers, except we have it on their own, own individual rows. Now, why would we want this? Well, let's say we want additional information, so let's just look at this one for now. So in this one, let's say we wanted to add additional things like the first name, so we'll do dm.first_name; we can do last or dem.last_name, so we can add other information, and it doesn't affect this column at all because we're using a window function. If we try to add these exact things, I'm going to go up here and do it; if we try to add these exact things to this, let's see if, um, yeah, that works, and then we also have to group by this; if we run this query now, it's going to be completely different because we're using a group by; we're grouping by the first name, the last name, and the gender; we're breaking everything out based off of the unique values in these columns, whereas down here it's completely independent of what's going on in these other columns; all we're doing is we're doing a window function just based off of that column. So I think that's pretty amazing, and there's a lot of additional functionality that we can do with these window functions, and we're going to take a look at a lot of those things in just a little bit. Let's try another example really quickly; let's literally just copy this, paste it down here, and all we're going to do is we're going to change this to sum. So now, instead of the average salary, we're looking at the sum of salaries, and we're still partitioning by the gender. Let's go ahead and run this, and let's pull this up, so all the men together make $42,000; all the females make $215,000. Now what we're about to do is something called a rolling total. If you've never heard of a rolling total, a rolling total is super cool; can be done within MySQL; a rolling total is going to start at a specific value and add on values from subsequent rows based off of your partition. So all we have to do is add an order by, and we're going to order by, let's say the employee_ID; let's go ahead and take a look at this, and it looks like the employee ID is ambiguous; I had a feeling, so I just need to say, uh, dem.employee_ID; let's try this one. So now we have something called a rolling total; I'm, I'm going to actually name it as rolling_total because this is super cool that window functions can do this, and this is something that a lot of people in like Finance do; I did it myself when I worked in healthcare, and it partitions based off the female, and you can't see the employee ID, but there's an employee ID that we're kind of ordering on in the background. Now what it's doing is it's starting with Leslie, nope, and she made 75,000, then the next person April, she made 25,000, which equals 100, and just to actually see this better, I'm going to add salary, and so Leslie no had 75,000, then we're adding this 25,000 to the 75, and we get 100, then we're adding the 60,000 to 160,000, then we're adding 55,000 to 215,000, so we're adding every single time; we're adding this salary to the already existing total all, all the way up to our grand total, which was 215,000; the exact same thing happens with the males, so we start with 50,000, then we add 50, then we add 90, then we add 70, all the way up to 42,000. Now you can do this in a lot of different configurations on a lot of different columns, but in essence, this is exactly what a rolling total is; that's how it works, and we were able to partition based off of this column; we don't have to use Partition by; we could do this completely regardless of the partition, but I, I thought it was interesting to at least break it out by female versus male. So now that we know how to use a window function, let's look at some special things that you can really only do with window functions or window-like functions. So we're going to bring this down, and what we're going to do is get rid of this entire thing, and we're going to look at something called row_number, then we're going to look at Rank, and then we'll look at dense_rank. So let's look at row_number, and this is just like an aggregate function like we doing the average age or average salary or something like that; this is what we're doing; we're doing a row number; now we're going to do this over, and we'll just do everything for right now. So let's go ahead and run this and just see what it looks like; let's bring it up, and what we're doing is we're saying, okay, we have first_name, last_name, gender, salary; that's all great, but then we get to row_number, and we're doing a row number based off of everything; it doesn't matter what it is, so we're starting at one, which is the very first row, and we go all the way down to the bottom just like an employee ID. So let's actually add that; let's do dm.employee_ID just like this. So we have this 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. Now, if you remember on this table, we are missing Ron Swanson, so it kind of skips that, but it's basically like an employee ID; we're kind of giving it its own unique value, and these row numbers are not going to repeat itself if you do it like this. Now they can repeat themselves if we do a partition, and let's do a partition on the gender again because we know how to do that one; we'll do Partition by, I spell that right, Partition by the gender. Now we're going to add a row number based off the gender, but again, it's broken out or partitioned by gender. Let's look at this; now it goes for the females 1, 2, 3, 4, then for the males it restarts 1, 2, 3, 4, 5, 6, 7. Now this is just in a random order based off how the, you know, data was stored in the table itself. Now, what if we wanted to kind of rank these based off of the highest salary first down to the lowest salary? You nailed it; we just add an order by, so we'll order by salary, and if you want to do it from highest to lowest, giving the highest salary the number one and the lowest salary, you know, later down, we'll do descending, and let's run this, and you'll see that for female we're still partitioning by gender; for female, the highest salary is one, next is two, three, and then four, then for males the highest salary is one all the way down to seven. So that's what row_number does; it just gives a row number based off of whatever you're partitioning by or ordering by in your window function. Now let's go over here and add a comma, and we're going to add, and let's go down just a hair; let's add rank, so we want to do Rank, and we'll do our parentheses. Now rank is going to give it more of an official Rank, and let's see how this works, so we'll do Rank, and we'll do over Partition by salary descended, the exact same thing, and while we're here, I'm going to rename these; I'm going to say as, as row_num, and we'll call this one, uh, rank_num. So let's go ahead and run this, and it looks very, very, very similar except for one small thing: this right here. So when we're using the row_number, whatever we are partitioning by, it's not going to have duplicate rows within that partition; it just won't. So even if there's 50,000 right here, it's just going to automatically assign it based off of something that is running in the background, whether it's the order of how the data is stored in the table or some other order by that you are using on the table. Now rank is a little bit different because rank is going to take it just like it did the row_num, except when it encounters a duplicate based off of order by, which is the salary, it's going to assign it the same number, so this is five and five. What's unique about rank is that the next number is not going to be the next number numerically; it's going to be the next number positionally, so this is 1, 2, 3, 4, 5; this is kind of like a six, and then it goes to seven, so it skips number six. Now there's another one; let's copy this rank; there's another type of rank called dense_rank, and we'll do dense_rank. So we'll do dense_rank, and let's run this, and let's pull this up; there we go. Now dense_rank is ever so slightly different than Rank in the fact that when it gets down to duplicates, it's still going to duplicate them, so it's going to have a five and a five, but it's going to give the next number numerically, not positionally; that is the only real difference between Rank and dense_rank, and again, row_number is just not going to have duplicates; it's going to give it its own unique within that partition. So I know I just threw a lot at you, but that's row_number, rank, and dense_rank in a nutshell, and you can review this, mess around with it; all of these things because, you know, these are actually really, really useful. So that's all we're going to take a look at in this window functions lesson; I hope all that made sense; I hope you kind of got an understanding of how it can work and how powerful these window functions can be, and this is actually the last lesson in the intermediate MySQL series. Thank you, thank you guys so much for watching; I really appreciate it; if you like this video, be sure to like and subscribe, and I'll see you in the next video.

[Music]

Hello everybody, and welcome to the first lesson in the advanced MySQL tutorial series. Today we're going to be looking at CTE. Now, CTE stands for Common Table Expression; they're going to allow you to define a subquery block that you can then reference within the main query. Now that may not make perfect sense, but we've looked at subqueries in the past or in previous lessons in the intermediate series, so you kind of understand that it's kind of like a query within a query, except we're going to name this subquery block, and it'll be a little bit more standardized, a little bit better formatted.

Than actually using a subquery, let's take a look at the basics of writing a CTE. Let's pull this down really quickly, and all we're going to do is we want to create this as a CTE, so we'll say "with," and that is our keyword to define our CTE. So we're going to say "with," and then we're going to name our CTE, and we'll just call it CTE_example, and then we're going to say "as." So this is how we define it, and now we need to actually put it in parentheses. Now you can do this in several different ways. I'm going to do it kind of like this, just to really emphasize that this is within the CTE.

Now CTEs are unique because you can only use the CTE immediately after you create it. So if we come right down here and we come right below it, we'll say "select everything," and we're going to say "from CTE_example." So we'll say "from CTE_example," and let's bring this back up. Now if we run this, we're going to get the exact same output. Now this should seem pretty familiar, almost like we're using a subquery, and within our subquery we have this right here; we're kind of building our own little table, and then we can query off of it down below. So we can come down here, and let's actually change the names in here. We're going to say "average_uncore_cell," and we'll change all of these real quick, just because I don't like having to actually put the, you know, the tick marks; I don't like doing that. So here we're going to say "Max," then we'll say "Min," and then we'll say "Count," and let's go ahead and run this again. And so now we have these different names, and when we come right here, we can say "select," and then we'll just do something really simple; we'll just do the average of average_uncore_c, so the average salary, and let's run this. And so this is the average between both the males and the females.

Kind of the purpose of these CTEs is to be able to perform more advanced calculations, something that you can't easily do or can't do at all within just one query. Another reason to use a CTE is just the readability. You can absolutely write this using a subquery, and let's do that really quickly, and it's just going to be a little bit tougher to read and look at. So let's come right up here; we're going to say "from," and we'll do right here; we'll say "select everything"; we'll do "select average_s from" here, and we're going to need to name this, so we'll say "do_example_undor_subquery." We get rid of this, and then we just need to get rid of this, and we can run this query, and we get the exact same output. Now if I formatted this exactly the same, just like this, and the names down there, if we look at this, the syntax is just a little bit more difficult to read. We're selecting the average of average_s from, and then we have our subquery right here, and then we're naming it at the bottom. If we scroll up and compare this, this one just looks a lot better.

Now when you're writing in MySQL, sometimes it doesn't matter if it looks pretty or not, as long as it gets the job done. That is true, especially if you're just going to be using it yourself, but in a more professional environment, when you're using this in your actual job, there are going to be people who have been using this for 10, 20 years, and they're going to expect you to write it well. They don't want it to be really messy; they aren't most likely going to want it to be written like this. I've been using it for quite a long time, and I much prefer CTEs over subqueries, just visually, and it makes it a lot quicker to actually read through. So that is just one of the reasons, although you get the exact same output. Now there is some additional functionality within CTEs as well.

Now one thing that I mentioned just a second ago is that when you build a CTE, you can only use it immediately after; you can't use it right below it. So let's go ahead and let's copy this query, and we're going to bring it right here. If we try to run this, and let's do this, we're going to get an error, and let's pull this up. It says "table Parks_andre.CTE_example doesn't exist." So we're looking for a table called CTE_example in our database, but it's not there. Now the reason this happens is because you're creating a CTE; you're not creating a permanent object like a temp table, which we'll look at in the next lesson, and you're not creating a real table, and you're not creating a view; you're really not creating anything. It's just a common table expression to create this table right here; this basically, almost like a temporary table, almost, but then you're just using it to query off of it; you're not saving it; you're not storing it in memory; you're not really doing anything with it; it's just like writing a regular query. So this is why you can only write it immediately after creating the CTE; you can't write it down below and reuse it because it's just like calling a query that you wrote before; it just isn't going to work.

Now the next thing that I want to take a look at, and let's copy this down here; the next thing I want to take a look at is that you can actually create multiple CTEs within just one. So if we wanted to do a more complex query or joining more complex queries together, we can do that all within one CTE. So let's come right here; let's get rid of all of this, and we're going to say "from the demographics table," we're going to say "where birth_date" and let's just do "is larger than 1985." D1, D1. So we have one query, and we'll take just a few columns from this table, so we'll take, let's say, the employer or employee ID, we'll take the gender, and the birth_date. So this is one query, and we're filtering just based off of this birth date. Now when we create this, this is the CTE_example, but we can have a comma here; we come down below, and then we can say "CTE_example_2," and I need to combine that, so "2," and then we can say "as," and then we have another query. So then right here we could say "select everything"; we'll change that in a second, "from employee," say "salary," and in the salary we'll just do a simple one; we'll do "where salary is greater than 50,000," and we'll actually just take the employee ID and the salary.

Now if I come right down here, I can say "select everything from CTE_example," which is—and let me scroll up so we can see everything—that's our original; our CTE_example is this first query right here, then we're creating our second one right here, and we can join basically on these two common table expressions. So now we can say "join," then we'll do "CTE_example_2," and I need to change that X, and then we'll say "on," then we're just going to do "CTE_example.employee_ID is equal to CTE_example_2.employeeid," and not an equal sign, but a dot, there we go. Now if we run this, it should work, and we can pull this down, down, and look at our output. Now this is just an example; this isn't a real use case because, of course, we just join these two tables together normally, but you can imagine you have a much more complex query or you're doing a lot of functionality within this table, and you just want a certain subsection of this table, and you're wanting to combine those; this is how you can do that with a CTE. So now we have all of our information right here, and that can be extremely, extremely helpful.

Now one last thing that I want to show you; we're going to go all the way back up really quickly, right here. Let's run this one one more time, and let's actually take everything, and let's run this. So here we have our gender, average salary, Max salary, Min salary, and count salary. The last thing that I want to show you, and this is more of something that's just somewhat helpful; you don't have to actually do it in your main query, is before we went in here and we changed all the column names by doing an alias by saying "as," and then saying the average salary, and the "as" is just implied here, but we're changing this via an alias; we don't have to do this. In fact, we could come right here, and we could do a parenthesis; we could call it "gender," we could call it "average_unor_salary," "max_core_salary," "min_ore_salary," and let's do "count_uncore_salary." So now if we were to run this, let's change this up; we'll do capital on this one. If we wanted to run it like this, when we run this, it'll change all of those names to what we have it right here. So this will be the default; this will overwrite the column names that you have in your actual CTE expression or the query that you have within your CTE.

So that is all we're going to take a look at in this lesson on CTEs. These are very, very helpful; definitely help with more complex queries, and they're just really easy to read and understand, which is why I personally use them a lot. In the next lesson, we're going to be taking a look at temp tables, and we'll also compare temp tables to CTEs, and we'll take a look at the [Music] difference. Hello everybody. In this lesson, we're going to be taking a look at temporary tables. Now temporary tables are tables that are only visible to the session that they're created in. So if I create a temp table right now and I exit out of my SQL and I come back in, it's not going to be there anymore, and we'll look at that in just a little bit. Now temporary tables can be used for a lot of things, but how I've mostly used them, especially as a data analyst, is for storing intermediate results for complex queries, somewhat like a CTE, but also for using it to manipulate data before I insert it into a more permanent table. So let's take a look at how we can create a temp table. There are two ways that you can do it. I'll show you the first way, which I don't think is as popular, and then I'll show you the second way, which is how I typically use it the most.

Now the first way to create a temp table is to create a temporary table. I need to sound it out like that; it's the only way I can spell it. Uh, so we're going to do "temporary table." So this is our name. Now if we just took this out and we created a table, this would create a table in our Parks and Recreation database, but we don't want that; we don't want to create a temporary table that just lives inside of our memory or the memory within our computer. Now we're going to create this temporary table much like we would a regular table, and we're going to need to name the columns as well as the data types. So let's do "first_name," and our data type can be "varchar," let's say 50, then we'll do a comma, then we'll do "last_name"; we're going to keep this really simple; we'll do "varchar 50" again, and then for our last one, we'll do "favorite_movie," and for this one it needs to be longer, so we'll do "varchar," let's say 100. Now let's get rid of this, and let's actually run this after we do our semicolon. Let's actually run this, and nothing's going to happen. Let's click refresh; nothing's going to happen, at least you can't see it happening. Let's pull this up, and you can see that "create temporary table say Z is affected," but it was created. Now in order to actually see it, we can do "select everything," and we'll do this from our temp table, and we'll add a semicolon, and then we run this, and we have this empty table right here.

Now what's really great about these temp tables is then you can insert data into it, and it basically is like a real table except it just lives in memory, and they go away after a while, but you can reuse this temp table over and over and over again. Now let's insert some data into here, and then we'll take a look at this again. So let's come right down here; let's insert data; we'll do "insert into," and we want to insert that into the temp table, and we're just going to say "values." Now we just say "values"; I'll use myself for this one; we'll do "Alex Freeberg," and what's my favorite movie? Give me a comma; that'll be "Lord of," I think it's like that, "Lord of the Rings: The Two Towers"; it's probably my favorite movie of all time. Now let's go ahead and insert this data, and let's pull this down here, and let's run it all the way down here after we add our semicolon. And when we run this, you'll notice that now we have data in here. So now we can use this table much like any real table. So that's the first way to create a temp table; not my personal favorite way, although there have been some use cases where I've done it like that. I'm going to show you the way that I typically do it, and for this, let's "select everything from the employee_salary table." Let's run this. Now let's say I just wanted a subsection of this data to sit in this temp table where the salary is greater than, let's say, 50,000. I could easily, easily do this. I'm going to say "create temporary table," and let's do this one as "salary_over_50k."

Now one thing about naming either temp tables or CTEs or subqueries or any of these things where you need to name something, I try to typically name it something that actually makes sense. So the "salary_over_50k" is something I would actually name it in my real work; I wouldn't normally name it something like "temp table." The reason for that is because when you're in a work environment and you have lots of temp tables, you're creating really advanced stored procedures, really advanced queries; you have hundreds or even thousands of tables and different databases; it gets really complex, so naming conventions are actually pretty important, or they become more important, the more you get entrenched in this stuff. So just something to think about. Now we're creating this temp table; now we don't have to really insert data into it more than we're just going to select data from an already existing table. So I'm going to say "select everything from," I'm going to say "employee_salary," then we're just going to say "where the salary is greater than 50,000." Now I want Tom and Jerry; I want them to be included as well, so I'll actually say "greater than or equal to." So now we're creating a temporary table based off of an already existing table, and we're just selecting data into this temporary table. So when we run this, now we can "select salary_over_50k," and let's run this, and it works perfectly. Now the great thing about temp tables is they last as long as you are within that session, meaning if I copy this query, let's go to a new window, and let's paste this in here. Let's zoom in a little bit, and let's run this; it still works, even in a new window, but if I'm to exit out and come back in, then it is no longer going to be working. Now let's exit out of this; let's come back in, and we'll see if these temp tables still work. Let's go ahead and exit out. Oh jeez, I'm embarrassed. All right, let's go to MySQL; let's come over here to the local instance. So now it pulls right back up. Zoom in once again on both these, and let's try to pull up our "salary_over_50k" temporary table. Let's run this, and we're not getting an output. Let's go back; it's going to say "error code: the table salary_over_50k does not exist." So it only lasted as long as we were within this session. So that is how we create our temp tables, and that's how we use our temp tables.

Now in the last lesson we had looked at CTEs. CTEs and temp tables both have their own use cases within MySQL. For temp tables, this is usually for the more advanced things, so I'm usually using these in stored procedures when I'm really manipulating data, and I'm doing a lot more complex queries overall, and often times I'll use multiple temp tables, and I'm joining them together, and I'm just doing a lot of more advanced stuff. With CTEs, it's typically more simple things because you can't make as advanced CTEs or as complex CTEs. So with those, I'm usually keeping it to just one level of transformation; I have my base CTE or my base subquery or query, however you want to call that, and I'm changing it or doing one, one level of advanced thing on top of that query; that's what a CTE is really great for. Temp tables, you can just get a lot more advanced with it; they also last within the session, and if I'm using it multiple times throughout something like a stored procedure, then it makes so much sense to use a temporary table. So this has been our lesson on temporary tables. In the next lesson, we're going to be taking a look at string functions. [Music] Hello everybody. In this lesson, we're going to be taking a look at stored procedures. Stored procedures are a way to save your SQL code that you can reuse over and over again. When you save it, you can call that stored procedure, and it's going to execute all the code that you wrote within your stored procedure. It's really helpful for storing complex queries, simplifying repetitive code, and just enhancing performance overall. So let's take a look at how we can create a stored procedure. Now we're going to start by just creating a really simple query; we'll make it a little bit more advanced as we go along and take a look at the different things within stored procedures that you can do. Now let's change this query; let's say "where the salary is greater than," let's do "50,000"; let's actually do "greater than or equal to 50,000"; we want to include Tom and Jerry as well. So let's go ahead and run this. Now what we want to do is save this really complex code within a stored procedure. Let's come right down here, and we can create a super, super, super simple stored procedure by just saying "create procedure," and pasting that. Now we just have to name it, so we have "create procedure," and we'll call this "large_salaries," and then we do a closed parenthesis. Now this is as simple as it can possibly be; it does not get any simpler than this. So let's go ahead and run this, and if we go down, we pull this up, you can see that it says "create procedure zero R affected"; it looks like it worked, and if we come over here to this refresh button, you should see now that under stored procedures it drops down, and we have our "large_salaries"; that's exactly what should have happened; we wanted to save that into our Parks and Recreation. Now if you wanted to be careful, you could say "use parks_and_recreation"; this is not a bad idea, but you don't have to, but you can specify what database within your actual editor window; sometimes that is helpful, but now we've created it. Now let's see how we can call it. All we have to do is say "call"; we're going to copy this entire thing, including the parenthesis, and let's end it with—that's right—a semicolon. Let's go ahead and run this, and as you can see, it worked because we got the exact output. So we actually called this stored procedure, and this code ran. So it's just a select statement, so it worked perfectly. Now you can also come over here to "large_salaries," and there's this little tiny, little button here that looks like a lightning bolt, and if you click it, it's going to open up a different window, and we'll say "call Parks_and_Recreation.large_salaries." So you can do it that way as well, but we're not going to be doing it that way.

Now what we've written right here is not best practice by any means, and I'm going to copy this down here because there's a lot of different things that you need to take into account when you're creating a stored procedure. For example, this right here is most likely not what you're going to be putting into a stored procedure; this is super, super simple. Typically you'll be having multiple queries, and let's see what happens if I try to put another query in here, and let's get rid of this. So we're going to "select everything where the salary is greater than 50,000," then we'll "select everything where it's greater than 10,000," which is everybody. Let's

Call this large salaries 2. So we have two different statements in here, and we want them all to be under this large salaries. Let's select everything and let's run this. We're getting an output, which is already not a good sign. But we created the store procedure, and then we selected everything, so what's actually happening here? Pull this back down. What's happening is is this is creating the store procedure, and this is just some other, you know, random query. But that's not what we want. What we want is everything, or both of these queries, within one store procedure.

The best practice is to use something called a delimiter. Now, this right here is a delimiter, the semicolon. So this semicolon separates our queries from one another. It tells MySQL, hey, you know, this is a different query; don't be mixing these and causing errors. You know, that's essentially what a delimiter does. Now we can change the delimiter by coming up here and saying delimiter, and we can change it to almost anything we want. Now, in my actual job, I've seen it done many different ways. I've seen these forward slashes, also seen dollar signs. This is probably the one that I've seen the most when I worked with data engineers, data scientists, database developers. You this one I see a lot. And then you'll come into the code, and you'll say begin, and let's go over here and let's tab all of this, and then we'll say end.

Now, when we end, we're going to end it with this dollar sign. So here's what's happening: We're changing the delimiter right here to dollar sign. We're creating our store procedure, and within it we are keeping all of this. So all of this code is going to go into this one store procedure. Then, at the end, we are saying this is the end right here of this store procedure. These semicolons no longer are the delimiter that's telling us when it is the end of the store procedure; that's what the delimiter does. Now, it is best practice at the end to change it back, right? Uh, let me spell it right, because if you don't, then you're going to have to start using uh these dollar signs for everything. And how do you spell delimiter? Oh man, there we go. Now we've changed it back to a semicolon afterward, so then we can go and write other queries, and it'll act appropriately. Uh, let's go down. So this is getting closer to best practice. Let's go ahead and run this entire thing. And if we pull this up, we're not getting an output; that's a good sign. If we pull this up, it's saying we already created number two. Change that to three. My APIs. Let's go down here. Now we've created the store procedure three. Now let's go over here; we're going to right-click on this; we're going to say alter stored procedure. And now you can see that we have both of these queries within the store procedure. Let's get rid of this, and we're going to go and call this. So let's copy this large salaries three, bring this all the way down, and let's say call that's St procedure. If we run it, you'll notice we get two outputs. We have six and seven. This result six is where it's greater than 50,000 or 50,000 uh or greater. This one is where it's greater than 10,000, which is essentially the entire table.

Now, so far we've done everything just by writing it all out, and that's fantastic. But you can't also come over here to store procedures; you can right-click and say create store procedure. Now let's actually copy this; we're just going to create the exact same thing. We'll create St procedure, and we can just paste this in here. And let's go ahead and do that. There we go. And sure, we'll call it new procedure, why not? And if we say apply, you'll notice that it generates this script right here, and we can apply it, and we can create it. We will in just a second, but let's take a look at it. So we're going to use parks and recreation; that's what I was mentioning before. We're then going to say drop procedure if exists. Now this is something that I was going to show you later, but I'll just show it to you now. Sometimes it is really beneficial to write something like this before you create it, in case you've already created a store procedure with that name that you're wanting to replace. So it's checking if it's there, and if that new procedure is already there, it's just going to drop it. Then it comes down, and let me see if I can zoom in on this, and then it's going to create our delimiter, which it uses dollar signs. So MySQL is even, you know, validating what I was saying earlier. We're going to use use parks and recreation again. And now again we have to use, instead of a semicolon, we're using dollar signs. Then we're creating the procedure, which is new procedure. We're saying begin end, and then it's even changing the delimiter back. So basically everything that I said, this is kind of doing it for you automatically. Now, when I click apply, it went ahead and executed that SQL statement, and our new one is ready. So we can go ahead and alter that store procedure, and it looks exactly the same as this one out here, which was uh large salaries number three. So it looks exactly the same. Now let's go ahead and get rid of this, get rid of this, and let's go down below.

The next thing I want to take a look at is something called a parameter. Now, before I actually get into this, I'm going to copy all this down here because I don't want to rewrite all of it if I'm being honest. So let's paste this in here. Now parameters are variables that are passed as an input into a store procedure, and they allow the store procedure to accept an input value and place it into your code. Let's take a look at what that actually means. Now, before I do anything, I'm just going to change this to uh number four, so I don't forget. So let's get rid of all of this; we're going to keep it somewhat simple because we're looking at something new. Now, when I say we're passing through a parameter, I'm talking about when we're calling it. So let's say we've already created this one; I'm not going to, you know, run this yet, but let's say we've created it. Let's say I want to pass in an employee ID. I want to pass in a specific person, and I want to retrieve their salary. I know their employee IDs; I just want it to pull up their salary for us. So what we're going to do is we'll get rid of this, and when we're calling it, put this down. When we're calling it, I'm going to pass through a value like one, that's Leslie nope, and then I want the salary to be the output. So I'm going to select the salary. So we're selecting salary from the employee salary, but how do we know that this one is the person we're looking for? Well, when we're actually creating this parameter, we create it right in here; that's what tells the store procedure to accept an input value. When we're calling it down below, we're going to call this employeeid. Now, after we call it, after we name our parameter, we need to then give it a data type. So I'll call this an integer. So we're telling the store procedure, when somebody calls the store procedure, they have to pass through an integer; it can't be a string, or it can't be a date; it has to be an integer. Now what we're going to go do is right down here; we'll say where the employee unor ID, that's from this column in the actual table, we'll say is equal to the employee uncore ID, which is our parameter right here. Now you may be thinking that's really confusing; they're named the exact same thing. Can I change it? The answer is yes; I actually encourage it. So there are some naming conventions that are out there that I think are helpful, ones that I personally use, um, but remember this is just kind of a variable parameter name; you can kind of call it whatever you want. So if I wanted to say Huggy muffin, I could, uh, and this could be Huggy muffin. So let's try it with Huggy muffin; I just came up with that off the top of my head, so don't judge me, um, but we're going to create the store procedure, and then when we call it later, we want it to return the salary where the employee ID right here is equal to whatever was passed through that parameter, that input parameter, and we we're going to keep it as one, so it should return 75,000. Let's go ahead; we're going to create this, and now let's go right down here, and we're going to run it, and we can see that that is the salary, and it worked perfectly. Now, like I was saying, that is not what I would actually name it. Uh, there are some naming conventions like underscore param at the end, so you kind of want to keep it, at least I recommend you try to keep it similar to what you're actually looking for, and you can either end it in underscore param, or there's another way you can do it, which is come right over here and do pcore, and these are just ways that you can tell the code, or you can just be able to visually see the difference in the code. So this is just what I recommend. Then you put it right down here; you say where the employee ID is equal to pcore employee ID, saying this is the parameter that's being passed through and put into our actual query. So that is all we're going to take a look at in this lesson. In the next lesson, we're going to be taking a look at triggers and [Music] events.

Hello everybody. In this lesson, we're going to be taking a look at triggers and events. A trigger is a block of code that executes automatically when an event takes place on a specific table. For example, let's take a look at these two tables. Now, when a new employee is hired, they're put into this table with their salary information and everything, but sometimes people forget or don't add their information, like uh, you know, who right here, uh, they're not put into this demographics table, and we want to change that because we want to have everybody in here. So when somebody is put into this salary table, we want it to automatically update with the employee ID, first name, and last name into this table right here for the employee ID, first name, and last name. So we're going to write a trigger. When data is updated into the salary, it's going to also update the employee demographics for us. Now let's go right down here, and we're going to take a look at how we can do that. Now, if you watch the last lesson on store procedures, we'll do a lot of the same writing style or same formatting for triggers and events. So we're going to start with is the delimiter; we're just going to do that right off the bat before we get into anything, and we're going to change that to the double dollar sign. Now the delimiter again, in case we have multiple lines of code, which we're going to have, if we have multiple lines of code when we're creating this trigger, this delimiter is going to help us have multiple queries within our create trigger statement. So this is really important; we'll just start out by doing that. Now let's create our trigger, and we do need to name this, so we'll say employee unor insert, and we'll just call it like that. Did I spell that right? Yeah, employee insert. So we have our create trigger; we've named it. Now we need to specify what event needs to take place in order for this to be triggered. So we're going to say after an insert, and I need to spell insert right, after an insert on, and we'll do the employee salary table. So after we insert onto the employee salary table, down below we're going to write what's actually going to happen. Now we're writing after because we're doing it where when new information is put on the salary table, it's automatically updated into the demographics table, but you could also do before, which means if data is deleted from the employee salary table, something could happen, but we're not doing any deleting or any updating; we're doing insertion. So we're going to say after an insert on. Now the next part that we need to write is for each row. Now this for each row means that the trigger is going to get activated for each row that is inserted. So if we had an insert statement that inserted four different people who were just hired, that means this trigger is going to be activated four times. Now some SQL databases like Microsoft SQL Server have things like batch triggers or table level triggers that'll only trigger once for all four of them, and in my opinion, those are really really nice. Uh, I've used those; I like them. The way that MySQL has it right here is not the most optimal way to do it, unfortunately, but we don't have access to the batch level, the table level triggers at this time. So this is really just the setup for what we're about to write. So after it's inserted on the employee Sal table for each row, what is going to happen? We're going to go down here, and we're going to say begin, and we'll have end. Now the code that we're going to write here is what's going to happen after this event takes place. So what we're going to do is we want to take from this table; let's bring this back up real quick. When we insert a new person, we want to take the employee ID, the first name, and the last name and automatically put it into the demographics table table. So we want to say insert, and let me do tab, insert into; we're going to insert into the employee unor demographics table, and we're not taking everything, so let's actually specify what columns we're doing. We're doing employee uncore ID, first_name, and then the lastore name. Now we need to specify what the values are. Now from the employee salary table, we're taking employee ID, first name, and last name, but we don't want to take all of them right; we don't want to take every single employee ID, every single first name, every single last name; we only want to take the new values that were just inserted. Well, lucky for us, there is something that we have for this. So let's do values new parentheses; we have something called new. Now new is going to say we're only taking the new rows that were inserted; there's also an old like this where it takes rows that were deleted or updated, but of course, for us, we're going to be using new. So we'll say new.employee ID, new.1store name, and then new.lastore name, and we'll close that. Then we'll come down here, and we'll do our delimiter, and we'll change this as well; we'll say delimiter back to a semicolon. Now we're getting this error because we need this right here. So let's recap what we've created; then we'll actually create it and try it out. So we're creating our trigger called employee insert. After a row is inserted into the employee salary table, for each row, here's what's going to happen: We are going to insert into the employee demographics table the employee ID, the first name, and the last name; those are the columns that we're going to insert into. Then we're taking the values new.employee ID, new.first name, and new.last name. Now MySQL understands that when when we say new, we're talking about the event that takes place; this is the data that's being inserted; it just knows that. So let's go ahead and create it; we're going to run this, and it should work. Let's pull this up; says create trigger, so that worked. Now the thing about triggers, uh, that's unfortunate; it doesn't have its own little section under here right, but it does have under the employee salary. Let's go right here, and then under the triggers, so we can find it, which is great. So we have this employee insert; if we right-click, we can't really do anything with it; that's the unfortunate thing; we can't alter it; we can't change it; we can't drop it; we can't do anything, um, that's the unfortunate part. But let's actually test it. So now we're going to say insert into, and we're going to insert into this employee salary; that's how we're going to trigger it. So insert into the employee salary, and then we'll do employee uncore ID; these are all the columns, the first_name, lastore name, occupation, uh, bear with me for a second, then we have salary, and then Department ID. So this is what we're inserting into. Now we have to do our values. Now this should be shorter, hopefully. We'll do 13; we'll call him Jean ralphio; there we go. Last name is sapperstein, just like that, and not actually just like that; that's not spelled right. So we have saprin; his occupation is entertainment; 720 CEO; how much is he making? Uh, let's say a million; is that a million? A million; he's making a million dollars, and he's really not part of any uh department, so we're just going to have null. So what we're about to do is we're only inserting on the employee salary table, but we're putting all the values that we need into the appropriate places. Let's add a semicolon; let's go ahead and run this; make sure worked; it says insert into, and then one row affected. Now let's come back up; let's look at our salary table first and get rid of this. If we pull this up, you can see Jean ralphio saprin; he was added. Let's go over to the demographics; is the moment of truth; let's see if it worked, and as you can see, it worked perfectly. We have Jean ralphio sapin. Now they do need to come back and fill in this information, but it's already in here, kind of queuing them up, saying, hey, we need this person's age, gender, birth date, all that other information. So that is how we can create a trigger based off of a specific table, and then when it happens, it just automatically does it for us; we don't have to really think about it; we just know that we've created a trigger, and we can actually go and insert data on that table; that trigger is going to work; it's going to do what it's supposed to do, and that's really really helpful in the real world when you're working with a ton of tables; a ton of things need to be automatically done, and you don't want to have to manually do this. So having these triggers can save you a ton of time. Now let's scroll down, and we're going to take a look at events. Now event is kind of similar to a trigger; a trigger happens when an event takes place, whereas an event takes place when it's scheduled. So this is more of a scheduled automator rather than a trigger that happens when an event takes place. These can be fantastic for a lot of things, like when you're importing data; you can pull data from a specific spefic file path on a schedule; you can build reports that are exported to a file on a schedule; you can do it daily, weekly, monthly, yearly, really whatever you'd like; it's just super helpful for automation in general. Now let's say the pwy council comes up with some new legislation; they need to save some money, especially in the Parks and rec department; we're just spending too much, or they're spending too much, and what they want to do is retire people who are over the age of 60 immediately and give them lifetime pay. So what we want to do is create an event that checks it, let's say every month or every day, and then if they're over a specific age, we are then going to delete them from the table, and they will be retired. This is a fake example, so you know, go with it. So what we're going to do is come right down here; we'll select everything from employee demographics, and let's run this, and let's pull this up. So let's say if they are over the age of 60, which unfortunately is just Jerry G; like I don't make the rules, but if they're over the age of 60, they're going to be automatically retired. So let's come right over here; we're going to say create event, and we'll call this the delete and delete uncore retirees. Now before, when we were creating the trigger, we were saying based off of a specific event, but here we're going to schedule it; we're going to say on schedule, and then we're going to say every, and we could do one month; maybe we'll look every single month, but here we'll do let's do every 30 seconds; every 30 seconds. Now we're going to go down; we'll say do, and this is going to say here's what needs to happen every 30 seconds. So we'll say begin and end. Now what's going to happen every 30 seconds is we're just going to start with a select statement, and I'll just copy this; actually, we'll start

With the SELECT statement, then we'll update it to a DELETE statement. But we come right here; we'll say WHERE the age is greater than or equal to 60. So if we just run this query right here, that's only one person, as Jerry Gach.

Now, if we want to write this correctly, we'll do the delimiter; we'll have the dollar signs. We'll have the dollar sign right down here as well, and we'll say the delimiter back to a semicolon. Now, every 30 seconds, we don't want to select people who are that age; we want to delete. So let's go right here; we're going to change this to DELETE. Now we know it should be deleting the right person, and we're going to go ahead and create this event. Let's go ahead and run this, and let's make sure it was created properly. It looks like CREATE EVENT 0 is affected; this should be working. Let's go back up to the demographics table and let's run this, and let's pull this down and pull this up. And as you can see, unfortunately, Jerry Gach is no more. Um, you know, he's just too old, and the Pony Council, they recognize that. And so it wasn't my rule; that was unfortunately Pony Council's Rule.

Now, really quickly, if that did not work, let's say you couldn't create your event at all. Let's go down here; we're going to show variables, and we'll run it just like this. Um, I'm going to show you how you may need to fix this. So we can say WHERE variables IS LIKE, and then we'll say, uh, event, do it just like this. So I have event scheduler WHERE the value is ON. If yours is OFF, which sometimes that can happen, you're just going to update this to ON.

Now, another issue could have happened, and I just want to explore this for just one second: you may not have permissions to delete things. If you do not come right up here, let's try to figure this out together. It's actually EDIT REFERENCES, and I want to say it's right here under the SQL editor at the very bottom, yep. So SAVE UPDATES, REJECTS UPDATES, and DELETES with no restrictions; this needs to be unchecked. So go to Preferences, go to the SQL editor, down at the bottom, unclick this. If that didn't work… Now, if everything worked perfectly, you don't need to change a thing, but if it didn't, I just wanted to work through, you know, some, uh, troubleshooting that you may just have to Google or chat GPT or something to try to figure out. So that is how we can create an event in MySQL to run on a schedule. Now, typically, you wouldn't do it on every 30 seconds; you do something like every 1 month or every 1 year, or, you know, a longer time frame, but you get the picture of what we're trying to do. So that is how we create triggers and events, and this is also the end of the advanced MySQL series. If you made it this far, absolutely fantastic work! In the next two lessons, those are going to be our projects that we're going to work on for this series. We'll have a data cleaning project, and we'll have an exploratory data analysis project. Both of those are going to include a ton of things that we've looked at in this series and even some new things that we didn't look at in the actual lessons themselves. So thank you guys for watching. I hope you enjoyed this entire series. If you did, be sure to leave a like and a subscribe below. I will see you in those [Music] projects.

Hello everybody, and welcome to the very first project in the MySQL series. Today we're going to be focusing on data cleaning. Now, if you don't know what data cleaning is, it's basically where you get it in a more usable format. So you fix a lot of the issues in the raw data that when you start creating visualizations or start using it in your products, that the data is actually useful and there aren't a lot of issues with it. So that's really what data cleaning is.

Now, what we're about to do is create a database; we're going to import a data set. This is a real data set, and what we're going to do is we're going to clean the data. So I'm going to show you and walk you through all the steps in order to clean the data. The data set that we're going to be working with will be in the GitHub, so you can just go and download that. I'll have a link somewhere in the description, but let's get started. First thing we're going to do is create a new database; we'll go right over here to create a new schema, and we're just going to call this one—we'll do this is worldcore layoffs. So if you can't tell already, uh, we're going to do World Layoffs; that's the data set that we're going to be doing. We'll just click Apply, and that creates our World Layoffs right here. Now we're going to go into here; there are no tables. We're going to right-click on tables and go to Table Data Import Wizard. Now we haven't done this yet, uh, in this series; we haven't imported any data, but that's what we're doing here; we're going to show you how to import data. So we'll go ahead and click Browse, and as you can see right here, we have this layoffs data set. Let's open this up, and we're going to click Next, and we're going to create a new table; there's no existing table in this database. You can drop it if it exists, uh, if you'd like to; it doesn't matter; this is new. We're going to go ahead and select Next. Now, right here is where you configure import settings. Now MySQL is going to automatically assign a data type based off of the data in these columns, so we'll take a look at the data later. Now there is one thing that you can take a look at real quick: we have this DATE column. Now, in here, it assigned it as a TEXT; that's because of the format. We are going to import this as the raw data; we're not going to try to change anything in the import settings; we're just going to assume this is how the data was in the table. So I'm not going to change anything, although this may be something that you would want to change to something like a DATETIME and go and fix that, but we're going to import this as the raw data. Let's go ahead and select Next; we're going to import it; we just select Next. Now this could take a little bit, uh, so while this is importing, I'm just going to skip ahead; this should take just a few minutes to import.

All right, this just finished. Let's select Next, and we imported 2361 records. Let's go ahead and select Finish. We can get rid of this, and let's refresh this. Perfect; we have our layoffs table. So we'll select everything, and I'm going to go and double-click on the World Layoffs because I don't want to write out the whole thing every time. So we're going to say FROM layoffs, and let's see what we get. So let's take a look at the data that we're going to be working with in this data cleaning project. So this data set is layoffs from around the world, starting I think 2021, and we'll take a look at that in this DATE column later, but it has the company; so it has the company that did the layoffs; it has the location of where they are; what industry they are part of; how many they laid off; the percentage that they laid off, so the percentage of their company; the date; the stage, which refers to the stage that the company is in, whether it's a Series B, Post IPO, uh, they don't know; then there's the country; and then we have funds raised Millions. So we have a lot of information here, and in the next project we're going to be doing exploratory data analysis. So we're cleaning all of this data, and then in the next lesson we're going to actually dive into it and try to find trends and patterns and all these other things.

So what we are going to do is we're going to go through multiple steps. Step number one is we are going to try to remove duplicates if there are any; that is the first thing I typically do, especially if I know this data shouldn't have any duplicates or it'd be, you know, repetitive or unnecessary to have duplicates. The second thing is going to be to standardize the data; that just means that if there are issues with the data with spellings or things like that, we just want to standardize it to where it's all the same as it should be. Number three is we'll look at the null values or blank values, and there's a lot of null values in here; there's even a blank value right here, and we're going to see if we can populate that if we can, and there are times where you should; there are times where you shouldn't; I'll kind of walk through that as well. And lastly, we want—want to remove any columns and rows that aren't necessary, and there's a few different ways to do that, uh, this one is a little bit, you know, um, let me write this actually real quick: remove any columns. So I'm just going to say there are instances where you can do this; there are instances where you shouldn't do this. When you're working with massive data sets and you have a column that's, you know, completely irrelevant, completely blank, you don't have any ETL process that is required for it, um, you can get rid of it, and it can save you time when you're querying your data. Now, with that being said, uh, and we'll talk about this later, in the real workplace, often times you have processes that automatically import data from different data sources. If you remove a column from the raw data set, that's a big, big problem. So what we're going to do is something I would actually do in my real work, which is I would create some type of staging or raw data set. Let's say this one's our raw one, and we could have even called this layoffs_raw. We're going to create another one; we're going to create a table, so we'll say CREATE TABLE, and let's call this one layoffs_uncore_staging, and we literally just want to copy all of the data from the raw table into the staging table, so we can do that really quickly by just saying LIKE layoffs. And if we run this and we refresh, you'll see we have this staging database, and let's copy this here, we go, and we'll do layoffs_staging, and so now we have all of the columns, and all we have to do is insert the data. So we're just going to say INSERT, then we're going to say layoffs_staging, go right here, and we'll select everything from layoffs, and let's run this, and if we select the table, we now have all the data over, so super, super easy, and now we have these two different tables. Now, again, why do we do this? Is because we're about to change the staging database a lot. If we make some type of mistake, we want to have the raw data available. This does happen; this is something that you do in the real workplace because you're not going to work on the raw data; it just—you shouldn't do it; it's not best practice. So I'm going to show you what I would actually do in my, you know, like a real job. So that's what we're going to do. Now we're only going to be working off this staging database, and we can copy this and make different databases for different things, um, as long as we have our raw data, we can really do anything we want going forward, uh, and that's what we're going to do. So the number one thing we're going to look at is to make sure that we are removing duplicates; we want to make sure we don't have any duplicate data in here, and if so, we're going to get rid of it. Now, really quickly, if you did my Microsoft SQL Server project, we did something very similar, but we had an extra column over here that gave the unique row ID, which made it really easy to remove the duplicates. Here there is no identifying factor that's going to be easy for that, so I'm just going to tell you up front: removing these duplicates is not going to be easy, but we'll walk through it every step of the way. So what we can do is try and do something like a row number, and we'll match it against all of these columns, and then we'll see if there are any duplicates. Now, I'm just—we're starting off strong; okay, we're jumping into kind of some of the more advanced things; it does get actually easier as we go, but this is the actual order that I follow, so, uh, I'm going to keep it. So let's try to identify duplicates. So let's copy this; let's pull this down; do _staging; there we go. Now what we can do is we can do ROW_NUMBER, and we'll do that PARTITION BY basically; we could do every single one of these columns; that's kind of what we're doing. So what we can do is we can say everything, then we can do a comma, and we'll say ROWCOR_NUMBER, and would be just like this, and we're going to do this over, and we want to PARTITION BY all of these columns essentially; we could just do a few for now to see if we get any hits, and then we can look at that, but they're going to be multiple companies that have layoffs in the same location and industry, although their total laid off would probably be different; the date would probably be different. So if we do something like, uh, company, let's do industry, we will do total_uncore_ladore_off, comma percentage_laid_off, and then let's do DATE. Now I'm doing DATE with the backticks because DATE is a keyword in MySQL, so if we do it like this, it just really makes it easy. So we're going to PARTITION BY all of these things. So let's do PARTITION BY, and let's bring this down real quick. So I'm just going to say OVER PARTITION BY, and we're going to call this as ROR_NUM. Now let's try running this; let's see if it works. Really quickly, it's important, and over here you can see that we have our row number. Now these mostly are unique, and these all look unique; I'm not going to scroll through all of them, but we want to be able to filter on this, so we can filter WHERE the row number is greater than two; if it has two or above, that means there's duplicates; that means there's an issue. So let's go ahead, and we're going to take this; we'll put it into either a, uh, subquery or a CTE; I'll create a CTE for this, uh, CU; it's really easy. So we'll say WITH, then we'll do, uh, duplicate_CTE AS, then we'll just do our parentheses; we'll paste this in here; get rid of that right there; and now we're going to say SELECT everything FROM this duplicate_CTE, then we'll say WHERE ROWCOR_NUM is greater than one. Let's run this, and let's add a semicolon; let's run this, and you can see that these ones have duplicates; so these are our duplicates actually, and we want to get rid of these exact rows. Now, just to confirm that these are, uh, the duplicates, let's look at this one; I never heard of this company, um, but we'll take it really quick, and let's SELECT; we'll say WHERE company IS EQUAL TO, and we'll call this Oda. So let's run this, and it looks like the—these—no, no, no; these aren't duplicates; that's a good thing we checked; okay, because it looks like, um, these aren't the exact same, although they're very, very close; these technically are not duplicates. So I'm glad we checked this; we need to do this PARTITION BY over every single column; that's what I'm realizing. So we'll do company, comma location; I'm glad—I'm genuinely glad, you know, it's good to make mistakes, um, and figure things out as you go; it really is important. So location, industry, total_laid_off, percentage_laid_off, DATE, then we'll do stage, and then we'll do country, and then funds_raised_uncore_Millions. So we're changing the CTE to partition over everything. So now let's run this. Okay, Oda is not in there; that's the only one we checked, um, but let's look at Casper; I know this—these are the, um, aren't these the mattress people? Didn't know they had layoffs, for guys, um… All right, let's take a look, and it looks like this row and this row are duplicates; these are our duplicates. So we are going to want to remove only one of those; we don't want to remove all of those; we don't want to remove both; that would be bad. Now, in MySQL, it's a little bit trickier to remove things than it is in something like Microsoft SQL Server, PostgreSQL; um, they have different ways that they can delete rows. For example, in Microsoft SQL Server, we could literally identify these row numbers in the CTE and delete them from it, and it would delete it from the actual table. We can't do that in MySQL, and I'll show you, uh, let's actually copy this; we'll go like this, and we'll say, uh, let's say we want to delete these; we'll say DELETE FROM, so we're deleting this FROM WHERE the row number is, uh, greater than one. What am I writing right here? DELETE; there we go. So DELETE FROM this duplicate_CTE WHERE the row_num IS greater than one; that's all these duplicates; we want to remove them. Let's try to do this; let's run it; let's go down; if we look at the bottom, it says the target table duplicate_CTE of the DELETE is not updatable. So you can't—cannot update a CTE; a DELETE statement is like an UPDATE statement, um, essentially. So what we are going to do is we're going to do something a little bit different because this is how I would love to do it; that makes it super, super easy to remove duplicates, but that is not always the way that things happen in the real world. I think what we should do is take this right here, and let's run this; we should take this right here and put this into, let's say a staging two database, and then we can delete it because we can filter on these R_NUMs, and we can delete those which are equal to two. So it's essentially like, you know, creating some type of table and then, uh, just deleting the actual column. So we're—that's exactly what we're going to do. So it's essentially just creating another table that has this extra row and then deleting it WHERE that row is equal to two. So, you know, somewhat fairly straightforward, but, um, let's try it and let's see what happens. So we're going to come down here, and do is create our table, uh, let's try doing WITH here; let's, uh, let's copy to clipboard a CREATE statement; let's see if this works. Perfect; that's exactly what I wanted. Now all we're going to do is say we're creating the table layoffs_staging_two. Now this is a CREATE TABLE statement, and we're naming the columns, and then we're also assigning the data type. So we have all these things, but we want one more; let's do a comma, and we want to add rowcor_num, and I need underscore num, and that should be an INTEGER data type. So we'll just keep it just like this. Let's go ahead and copy this, and let's run it; see if it worked; bring this up; looks like it worked properly, uh, and let's say—let's go back up; I want to rewrite things that I don't have to—let's run this. So we now—we have this empty table. So we want to insert this information right here. So we're going to INSERT INTO; so we'll INSERT INTO, and then we'll do this right here. So INSERT INTO staging_two. Now let's try to run this; see if it works; and let's run it; and let's select that table; and now we have it. So let's pull this back up, and I'll walk through what we just did, cuz I know I'm going quick, but we have so much to cover, um, in this lesson. So—so we just inserted basically a copy of all these columns, but in this new table we added one more: the row_num. So now we can filter; we can say WHERE—I need to spell that right—WHERE row_num IS EQUAL TO two, or we should—should say greater than one, because some might have multiple duplicates, and there you go; here are our duplicates. Now we're going to delete these. So all we have to do is come right back down where…

I go copy this, come right back down here, and we're just going to say delete from. We just did a select statement. I always recommend doing that to identify what you're deleting. Then you change it to delete, and now if we run this, go. And I'm actually going to keep this um, let me see, there we go, and let's run it again. And now they're gone. And if we say um, just the whole table, this looks wonderful. Now this row numb is going to be a column at the end that we probably don't need anymore, right? It's a redundant column; it adds up extra space in memory and storage and all these other things and processing times. We're just going to get rid of it. Uh, that'll be at the very end, I'm sure. So it looks like we are good to go. That's how we remove duplicates. Now um, there are different ways to do it when you have different columns, like if you have a unique column over here, makes it so much easier, so so so much easier. But we didn't have that, so we had to kind of do a workaround. Uh, Welcome to the Real World.

Now let's look at standardizing data. So standardizing data is finding issues in your data and then fixing it. So I'm already noticing right here, looks like we have a space at the beginning. Uh, we could easily just do a trim on this column, um, and let's, I I don't even think I was um, I did this when I wrote out all the the scripts for this. Let's just do from this table. Why am I writing it all out again? We actually want to select the company and then the or actually we'll just do distinct company. Distinct company. Let's run this. And if we do a trim around this, let's run this again, and that looks better. So if we do uh company, company comma, and then we'll just do the trim, I don't want to, we don't need to do distinct right now. We'll do the company. This just looks better, so we're going to update that. Uh, it's super easy.

Now if you ran into just a second ago, uh, I may need to help you change that. So if you couldn't update or delete those things earlier, I should have told you this earlier, I apologize. All you need to go is to edit, you just need to go to edit, go to preferences at the very bottom, go to SQL editor, go all the way down to the bottom, and right here we have safe updates on. If you have this selected, that means you can't update anything; that's a problem. So what you need to do is select this uh, or unselect it like I have it and save it. You may have to even restart your MySQL potentially uh, in order for those changes to take effect, but then you should be able to update that. Now all we're going to do is update this table, and we're going to set, and now we need to come back here, and we'll say we're going to set the company equal to trim. Now if you don't know what trim is or you haven't taken that lesson, trim just takes off the white space, space off the end, so it took the white space out of here or off the right-hand side as well. So we're going to update this, and let's do a semicolon, a semicolon. Let's run this. Let's select this again, and it was updated properly. So we're already off to a great start.

Now the next thing that I want to take a look at is the actual industry. So let's go back, copy this, and let's take a look at the industry. So we'll do industry, and we'll run it. Now if you look in here, there's a ton of different industries um, and there's marketing and marketing, oh, because I haven't done distinct. Uh, please ignore me. Let's do distinct, and there's a ton of different industries in here: Transportation, Healthcare, consumer. Uh, there's a blank one which we'll take a look at, Aerospace. There's a lot of really unique ones. Let's actually order this; we'll do order by uh, and let's just do one, which is the first column. We're just ordering our s, so we have null, we have blank; that's a problem. We'll take a look at that later. Uh, but this is an issue: crypto, cryptocurrency, and cryptocurrency; these are all the same thing. These should all be uh, on or labeled the exact same thing. The reason we need to change this is because when we start doing uh, the exploratory data analysis, visualizing it, these would all be their own rows, their own unique thing, which we don't want. We want them all to be grouped together so we can accurately look at the data. Let's take a look at any other ones: fintech and finance; that could be the same thing. I'm not 100% sure; I'm not a fintech person. Um, I think for now the only one that I'm confident in changing is this one right here, which is cryptocurrency. So let's go ahead and update that. So all we have to do, and we need to actually, let's select really quickly where it's like crypto. So we'll say uh, where in industry and we want to select everything where the industry is like, and we'll just do crypto. They all start with crypto, right? Yeah, we'll do crypto just like this, and let's run this, and let's just take a look. Lot of layoffs in the crypto industry. Good night. All right, let's find where it's cryptocurrency. Okay, so even this one, it's crypto, and I know Gemini, crypto, crypto, and then it says cryptocurrency. So these should be all crypto. You see how 95% of them are crypto? So we're going to update these other ones. Oh, this one is CR YP; is that how you spell crypto? Jez, I don't know anything. All right, so we want to update all of them to be crypto. So what we're going to do is we're going to say update layoffs_industry_2, we want to set the industry equal to crypto just like this, where, and we can do it a few different ways. We can say industry, we, I think we can do like, let's try this real quick. I, I some of this stuff I don't have planned out; I'm just kind of going with it as we go, um, which I like better, you know, we kind of we work together on this, we figure these things out together; that's what I like. Um, then we'll do like crypto just like this, exactly like we had it up here. So if it's like crypto, it should be crypto. Let's try this; let's see if it ran because it may not have. I can't remember. Yeah, it worked. Okay, so it updated uh, three rows, and that looks correct. Now let's go back up and let's run this. As we scroll down, they are all the exact same. Beautiful, beautiful, beautiful. So if we do uh, distinct industry again, let's get rid of this. If we run this query and we scroll down, crypto is its own thing. Beautiful, and it looks great. We can look at those later on how we can update those, um, but let's keep going. Let's look at our whole table again, and these blanks and these nulls are actually an issue. We do need to deal with them, but I, I my instinct is telling me go fix it, um, but my, you know, tutorial side is saying, okay, stick with uh, the tutorial, the order that we agreed on. Um, so let's go take a look. We've looked at company, we've looked at industry. Um, let's just real quick look at uh, distinct uh, location. Now it's good to look at most of these things; right there could be small, tiny issues that you just never saw, um, and we're just going to order by, order by one, just do a real quick, just a scan to see if we find any issues. Um, that could be an issue, but that could just be another language if I'm being honest. I don't know. I'm as I'm just scrolling through here because I want to make sure because this is not something I had in my uh, pre-written script. This looks pretty good to me. Um, let's do everything. We'll run this, and now let's look at Country. So we'll do distinct country, and let's run this, and let's scroll down again. This is sometimes just what I actually do. All right, we got an issue right here. Super common. Somebody put a period at the end. Some dingus, uh, and we're not going to judge that person. I don't know who it was or who ruined this data set, but um, that's a problem. So we're going to need to just update that. It looks pretty simple, um, but I'll just say where country is equal to or let's say like, and then I'll say like United States. There we go, and oops, want to say select everything. I just want to see um, where it's at. Oh jeez, there's too many. Let me see if I can spot it. I can't spot it. It looks like they're supposed to be United States, not United States. That's the issue. Um, we can easily, easily fix this, and we can probably, let's do um, really quickly, let's do select, oops, select distinct, and then we'll do country comma, and then we'll do, do a trim cuz we want to get rid of that um, that one. We'll do country. Now just doing the trim won't fix it. Let's go to the bottom. So that doing the trim doesn't fix it, but here's what you can do. It's a little trick of the trade here. We're going to do something called trailing, which means coming at the end. So what's trailing the period from country? Let's try running this, scroll to the bottom, and it fixed it. So this is a little, a little advanced, little tidbit for the trim here. We can do trailing from the country, and we're looking for something that's not a white space. We're specifying we're looking for a period. So now we can do is we can say update, we can set the country. Do update um, this table, and we'll oops, and we'll set what am I doing? What's going on here? We'll set the country equal to, and we'll do it just like this, but we're only going to do it for count country, right? Uh, so we'll say is equal to trim, and we'll say where country is equal to or actually let's say like, and let me see if I have this. Oh, I don't. Let's let's just say like United States, like we had it before, just like this. So let's go ahead and update this after I put my semicolon in. Let's run this, and let's run this again. It shouldn't need to fix it anymore; it's just one row; that's perfect; that's exactly what we wanted.

Now one thing that's really important, uh, and this is, you know, this is a longitudinal, it's not the right word at all. Give me a second. I can't, I can't speak and write at the same time, so sometimes I just say uh, dumb things. Um, uh, if we want to do not longitudinal but um, time series, that's the word I'm looking for. If we're trying to do time series um, exploratory data analysis, Time series visualizations later on, this needs to be changed. Right now it's text, and we can look at that by going right. Actually, let's refresh this. We're not looking at staging; we're looking at staging two. If we look at the columns and we come down here to date, it is a text column; that's not good. If we're trying to do uh, time series stuff, we want to change this to a date column. Now how can we do that? Let's take a look. So let's do date backslash. So we're just going to look at the date. Now let's change this because we want to format it how we want to format it with it, which is month day year. So how can we do this? Well, there's something that's very, very helpful, works perfectly in this situation, and is exactly what we're going to do. It's called string to date. So we're going to do string underscore, there it is right there, underscore 2core date. It literally helps us go from a string, which is a text, that's the data type, to a date. So it's perfect. Now all we need to do is pass through two parameters. We have to pass through the column, which is the date column, and then what format we want it in. Now if you haven't done date formats before, I'm going to kind of walk you through it while we're looking at it, um, in order to format this properly, you use a percent sign. So it's going to be a formatting for a month, a lowercase m; a capital M is something completely different. I believe it's spelled out. I need to, we can look at that in a second if we want to. Actually, and then we can do this right here, and then we'll do another one. So we're formatting it in the way that we want it, but also converting it to an actual uh, date column. So now we want month, and then we want day, lowercase day. We'll do a forward slash, and then another percent sign, and then a capital Y, which stands for I believe the four um, four-number long year. Uh, have a, let's just um, let's look at this real quick. So it worked perfect. So we're, it's taking in this format that it's in right over here and converting it into the date format. So this is the standard date format that you're going to find in MySQL. Now let's see what happens really quickly, just for fun. Uh, let's see if we do capital M. Uh, it looks like that's not going to work at all. Uh, let's do lowercase Y, and um, just formatted it to 2020. I think it took the first two numbers. It looks like I don't know why it's doing that if I'm being honest, um, but if we keep it with the capital as we should, this looks perfect. This looks exactly like what we're trying to do. So you can mess around with it; it depends on the how the data is formatted in your original column when it converts it to the string to date, and there's a lot of different stuff you should just look up um, date formatting in MySQL. Really interesting stuff. So we're going to update this date column to this, which is our new date column. Let's go ahead and do that. We're going to say update. You guys should be getting used to this by now; that's the whole point is getting used to doing these things. So we're going to set date equal to, and then we're going to put in this right here, the string to date. Go ahead and do this, and let's run it. Make sure it worked. 2355 rows. It looked like it did every single one, uh, but let's go ahead and get rid of this, and let's run it, and it looks like it worked perfectly. Now there were some nulls it looks like, and that'll be something we have to look at later when we talk about nulls, but um, overall, I believe this looks proper. Now if we refresh this, let's refresh, let's come down to the date. You'll notice it is still a text; it's date, it's called text, but now it's in the date format. Now that's really important, and maybe I should have done that earlier if I'm being honest. Um, tried to convert it to a date column; it wouldn't work; it would give us an error. Um, you'll just have to trust me on that one, but now we can do it. We we can change it to a date column. So let's do alter table. Now only do this; never ever do this on your raw table. Only do this on things like a staging table because we're about to completely change the data type of the actual table. So we want to change the layoff_staging_2, and then we're going to come down here, and we're going to say modify column, and what column are we modifying? It's this date column. There we go, and we want to change it to what data type? A date. And am I spelling this right? Yeah, I just need a semicolon here. Whenever I see an error, I always, you gotta just look for the semicolons. So let's go and run this, and let's refresh, see if it worked, and the date was changed to a date, which is perfect. That's all we wanted to do. Uh, just to make sure we were doing what uh, or we'll set ourselves up later in the future really well. Let's look at our table. All right, this is very good. So we fixed a few uh, just issues with the company. I believe something with the industry. Oh, the cryptocurrency. We changed the country. Um, I'm just going to go ahead and tell you right now this one uh, we're not going to look at until we look at the um, nulls and whatnot in just a second. So we're not looking at that one yet, and then uh, we have this extra column that we've done. So we've done a lot so far, but the next thing in the process, step one was remove duplicates, step two was standardization, step three is working with null and blank values. Now this is going to happen; you're going to have nulls, and you're going to have uh, blank values in here. I, it's somewhere um, it's just going to happen, and so we need to think about what we're going to do with that information, whether we want to make them all nulls, make them all blanks, try to populate that data. Let's see what we're going to do. So let's start off with the total_laid_off. We'll just do uh, where total_laid_off is null. So in order to look at the null, we say is null. Let's try equal to null; it's not going to give it to us. We have to say where it is null. So we have these values; these are completely null. Uh, there's quite a few of them, but remember this is also useful information, but if they have two nulls, uh, that probably is pretty useless to us. Um, that's something I think we'll take a look at in a little bit. Actually, we'll say we, and we may save this query, percentage_laid_off is null. So if they're both null, like these, these are all I believe fairly useless to us. These might be ones that we remove. So let's actually look at this um, in step four when we look at removing rows and columns, but one thing we should take a look at, I remember this industry. Let's do uh, industry, do distinct. This industry had some missing values, and let's take a look at that. Okay, so we have a missing value, and we have a null here. So let's look at this query, and let's say where industry is null or do industry is equal to a blank like this. We'll select everything to run this. All right, so it looks like there are a few that are blank. Now what we can try to do is see if any of these have one that's populated. Let's take Airbnb for example. Let's search for this really quickly, and this is 100% um, you know, it's just helpful; it's really, really helpful to be able to populate data that is pop poppable; is that a word? Um, let's try it. So we'll say uh, select everything. I just wanted to do where, spell that right, where company is equal to, and let's do Airbnb. There we go. Let's run this, and it looks like we have this one right here. So for example, um, these, whether they have them or not, we're going to try to populate these. If this Bal or Carvana or Jewel had multiple layoffs, these ones should, if these ones aren't blank, if they have one that's not blank, we should be able to populate it. For example, um, not the one I was trying to do. If we look at Airbnb, this one has travel, so we know this is the travel industry, so we can populate this with travel. Again, we want this data to be uh, the same. So if we're trying to look at, you know, what industries were impacted the most, this row isn't going to be affected or this row won't be in our output because it's blank. We want that to be traveled to represent the data properly. So we want to update it. So if this one has travel, we should be able to update this row with this travel right here. So let's see how we can write this, and let me give myself some rows right here. All right. Now what we're going to need to do is try to do a join here. So let's try running out in a select statement, then we'll just change it to an update if it works. So we're going to select everything, and we're going to do this from staging_2, from staging_2, and we'll call this uh, St2, and then we'll join on itself, cuz what we're going to do is we're going to check in this table, does it have one that is blank and not blank? If so, update it with the nonblank one. That's essentially uh, in layman terms, what we're trying to write, but

Writing it out could be a little bit more difficult, um, so we're going to join on itself, and we'll call this uh, let's actually call this table one, T1 and T2, because they're the exact same table, uh, and we'll do this on, and we're going to say t1.compan is equal to t2.compan. So the company has to be the same; that's important. And we probably should do the location is the same as well. Now we'll do and t1.location is equal to t2.location. I'm imagining, you know, there's another Airbnb in like South America somewhere that's called Airbnb, but you know, I'm just imagining a scenario area right where we have to think about different use cases rather than just large companies. So those other ones, they may have ones that are in different locations; we don't want those. Um, we don't want to change them if they're not the same. So these are the same. Now what we want to find is we're going to say oops, we want to say where, then we'll do t1.industry is null, and then we want to check that t2.industry is not null. So we'll say and T2.industry is not null. And let's just run this; let's see if we get anything.

So let's think this through because we got nothing in our output. We're selecting everything; we're joining on the company and the company, um, and the location where T1 industry is null and T2 industry is not null. Let's just get rid of this for a second. I just want to see if this changes anything; it doesn't. And it's possible actually that instead of doing is null, we could do or, and this I'm glad we're walking through this, we can do or is equal to blank. And let's try running this. There we go. Okay, so it looks like there's Jewel, Carvana, and Airbnb; these ones all have Industries um where it's null or blank and an industry is not null. So that's really good. Now if we scroll over, see the industry here, this is our T1, this is our first table. If we scroll over, I bet we'll see the t2 industry where it's not null. Let's scroll over, and here's our industry; we have travel, transportation, and consumer. So this worked exactly as we had hoped. I can even um pull this up here just to show, kind of show you a little bit easier what that's doing, and we'll do t2.industry. This is kind of like what we're trying to do. So if it's blank, this one is going to be populated into here if there is one that is not not blank. So that's essentially what we're going to do.

Let's write the update statement, and we're going to see if it works. This we have to translate this to an update statement, so we'll do update, and we're going to update uh this right here, so we'll say update T1, and then we'll do the join right there, and now we have to do a set statement, so we'll set uh the t1.industry equal to, and I'll just copy this t2.industry. I just I don't like writing things out, um, then we say where, so we do this just like that, and let's add a semicolon. Okay, let's confirm. So we're updating this table T1; we're joining on T2 or the company is the exact same; we're setting T1 industry equal to T2 industry, so the T1 should be the blank one, so where the T1 industry is null or blank and T2 industry is not null. Let's go ahead and run this; semicolon. See if there were about three updated. Y REM match zero R affected, though. Let's go take a look. Um, we have to let's run this query. Looks like those are still null. Let's run this; that one is still blank. Now let me think here; I'm I'm trying to think of why this didn't work, and I want to walk you through my thought process. It is possible that because these are blanks and not nulls that it's not working. I and I will say that is something I typically do where I set these blanks to nulls first. So let's actually try that and see if that changes anything. I'm just going to update uh this; I'm going to say set the industry equal to null, and we'll say where industry is equal to blanks. So we're just changing it to null where it's blank. Let's try this, and let's go back down here to our select statement. So these are all nulls. Okay, I think I think this is now going to work because now you can see on this side it's going to; there's only one option for it to populate it. Before there were those um blanks, which I think was causing the issue. Um, let's get rid of this part cuz now we have no nulls, and now let's try running this. We're workshopping this on the fly, guys. Uh, let's see; three rows affected; heyo. All right, let's go see if it worked. Um, let's run this query, and we have none; that's perfect. Let's look at Airbnb. All right, all right.

Ran into some issues, but we worked through it; we figured out the issue, and now it's working properly, and we can even come back up here to select everything, and it looks like Bab Bailey is the only one that still has a null. Let's look up Bailey's real quick, and we'll say where company is like uh Bailey. Let's run this. Yeah, and there's only one, so there wasn't another row. All these other ones like Carvana and um I can't remember the other, Jewel and Airbnb, those ones had an extra row; they did multiple layoffs. This one only did one layoff, so we don't have another populated row where it's not null to actually populate the null row; that's really all that happened. Uh, that's why that worked that way. So I'm really happy that worked. Awesome job, guys. Uh, I was starting to question myself; do I even know how to use MySQL? I mean, I was really starting to question my abilities here. Um, take a look; I think that is all we're going to do for populating null values.

Now here's why things like total laid off, percentage laid off, um funds raised; how are we going to populate that with the data that we have here? I don't believe we can. Now we might be able to populate oops, we might be able to populate some of this if we had the um company total, like if we had the original total before laid off, because then we could do calculations like um oh, these companies went completely out of business; that's not good. At 1%, that means 100% was laid off, um, but if we had, you know, the total, they had 50 employees and 100% were laid off, we could populate the total laid off; whoops, did it again; we could populate the total laid off by saying if this is 50, 100% was laid off, that's 50 people were laid off. We don't have that data, so we can't go and populate it. I don't believe funds raised; we might be able to scrape some data from the web and populate this, but that's a totally different thing, um, not part of this project. So I think the data cleaning for the null values and blank values, I think that's going to be done. Um, it's possible that the stage could be the same, and if you want to go check, you can, but we're going to keep chugging along because we want to remove columns and rows that we need to. Now if you remember, we were looking at this before; did I save that uh query? Let's go look. Here we go; bring this down to the bottom. All right, these rows; let's let's really take a look at these um and think about if there's going to be health dust. Um, what we are trying to do with this data in the near future is we're not just trying to identify a company or a location that had layoffs, and maybe we are, maybe that maybe we are trying to do that, but these have no layoffs and no percentage laid off. So in my opinion, I don't know if these laid off any at all. Um, I believe that we can get rid of these. Now deleting data is a very interesting thing to do; you have to be confident. Am I 100% confident? No, not really, but I'm confident enough to know that what we're about to look at in the next one, we're going to be using these total laid off a lot, percentage laid off a lot when we're looking at um, you know, actually querying the data and doing some exploratory data analysis. So we're going to use these a lot. I don't think uh these; I'm not even sure if these are accurate; I'm not even sure if they actually did have a layoff; it's saying they did, but it doesn't show if they laid off any. So um, can we delete this? Yes. Should we delete this? It's iffy. Uh, I'm not 100% if I'm being completely honest, and there's a lot of rows like that. This is this could be like 100 or so, really not; I mean, I could run a query and run it, but I don't want to; I don't; it's not a big deal. The point being, I don't think we need this information, so we're going to get rid of it, if nothing else just to show that you can do it. So now we'll say uh Delete, and then we'll do from here. There we go. So now we're going to delete these rows. Let's try to select them again, and they are gone. So we deleted the ones where the total laid off was blank and the percentage laid off was blank. We just I can't trust that data; I really can't. Um, and let's go back down; I'm right here; a semicolon. So sometimes I have to walk myself through these things. Um, all right, this R num; I mean, come on, we don't need that anymore; let's get rid of it. Um, so what we can do now, it's a little bit different syntax; we want to drop a column from this table, so we have to do the alter table again. We're going to alter table layoff staging two, and then we're going to say drop column and rowcor num. If we run this, then we run the table again; it should be gone, and it is. So this is it; this is our finalized clean data. Now in the next project, we're going to be doing exploratory data analysis on this cleaned data; we're going to finding trends and patterns and running complex queries; it's going to be phenomenal. I'm super excited about it, and I love this data cleaning one. Um, I made some mistakes; I'll be the first one to admit, but cleaning data is not always a straightforward thing. Um, you know, you have to you kind of mess around with it, figure it out, and and you know, that's what we did. Uh, whoa, took a while. So just a recap, we removed duplicates, we standardized the data, we looked at the null values or blank values, then we removed any columns or rows. So we did a lot. Um, if you go back and you actually scroll through here and look at some of this code that we wrote, uh, it's not super beginner stuff. So if you're following along with these things and you are getting this project, this is a fantastic project to put on your portfolio. I myself would put this project on my portfolio because it's a very very relevant thing. So I hope this was helpful. I'm just going to keep scrolling the light talk. I hope this was helpful; I hope you learned something. We did a we did a lot of different things that we didn't even do in the lessons, which I like doing because you can't cover every single aspect of MySQL in lessons, right? Sometimes you just got to get in there, get into the nitty-gritty, clean some data, and you'll find uh or discover new things, try new things. Um, and now we're getting to the bottom, and awesome work; awesome awesome awesome work. Uh, this is an A1 project; I think this should be in everyone's portfolio. If I don't see it in your portfolio and you know you send it to me, I'm going to say it's the garbage portfolio. So this is a good one. So with that being said, thank you guys so much for watching. I if you made it all the way to the end, you're still listening to me; awesome work; really awesome work for real. I you know, you're just following along with the tutorial; that's what it feels like, but by the end of this, I just know you're learning a ton, and you're you're trying new things, and you're really pushing yourself beyond just simple tutorials. So trust me when I say this is not easy; not everyone was able to make it to the end, so great work getting here. So I will uh see you guys in the next project when we actually explore this data. We walked through a lot of different ways to do that. So thank you again for watching. If you like this, be sure to like and subscribe below. I put out tons of content about all this stuff, and I absolutely love it; it is definitely one of my passions in life. So go ahead and do that, and I will see you in the next video. [Music]

Hello everybody; in this project, we're going to be focusing on exploratory data analysis. Now in the first project, we worked with this exact data set, and we cleaned up the entire thing, and that was a really good project, and it set us up to explore the data, and with all that clean data, we'll be able to look at our data much better and find better insights while we are using it. Now normally when you start the EDA process or the exploratory data analysis process, you have some idea of what you're looking for; sometimes not always, and sometimes when you're exploring the data, you also find issues with the data that you then have to clean. So even though I did a data cleaning video and then an exploratory data analysis video, and they're kind of separate projects, sometimes those coincide together where you're exploring it and cleaning it at the same time. Now what we're going to be doing here with this data set, we're just going to be kind of exploring it; I don't have any agenda; I don't have any, you know, one thing that I want to look at; I just kind of want to look at everything, and we'll kind of discover and go uh about things as we are learning and looking at this data set. We will, however, start off really simple with kind of the basics, work a little bit more towards the tougher stuff, and then at the end, we'll have some more advanced things that I think will be really fun. So with that being said, let's start off with kind of more easier things; we'll kind of just ease our way into exploring this data set. Let's pull this down, and let's copy this right down here. Now we're going to be working with this total laid off and percentage laid off, or most likely this total laid off quite a bit; the percentage laid off isn't super helpful because we don't know how large the company is; we don't have another column here that says here's how many total employees they had, and then okay, they had a percentage laid off; you know, we won't work as much with this one, but we'll work quite a bit with this total laid off. Let's look real quick; we can look at something like the Max uh total, and I need to use a parenthesis, Max total laid off, and let's look at this. So on one day, there was somebody out there who had the Max total laid off of 12,000 people; that's a lot of people to lay off in one, you know, one go; that's a lot. U let's also take a look at the max, and I think it was percentage laid off. Let's run this, and it looks like one. Now one represents 100; that means 100% of the company was laid off, um, and that's, you know, that's not great; it just means an entire company went under essentially. We can actually take a look at that because I'm interested to see, you know, if there's any companies I recognize or can see, um, where and come right down here where the percentage laid off is equal to one. Let's go ahead and look at this, and let's take a look. So we have this ahead; I'm just going to go through here and see if I recognize any of these. Uh, some in the crypto space, BlockFi; I feel like I recognize that one; I don't know. Uh, let's keep going; Deliver; it's not good; they let go of 120 people. I'm just curious; I mean, I'm I'm just kind of scrolling through here trying to see if I recognize any. These are companies that like completely went under or or lost all their employees. Vault Bank; interesting; just interesting to me. We're going to be taking a look at a lot of stuff, um, but these are companies that completely went under, um, that's, you know, unfortunate. We can also order by uh total uncore laid off, and that's not how you spell it, in descending; we'll see which company Wonder had the largest. So this one had 2,000; construction company had 2,400 people; they went um under; doesn't say what stage they were at, but that's in the United States. We can also take a look at, and there's another column over here called funds raised in millions; let's look at that one. So I want to see um these are companies that had a lot of funding or potentially a ton of funding. Uh, let's go over; this is like $2.4 billion, I believe, like I think is like a ton of money, um, Quibby; I believe I know this company, uh, in Blocky; I I thought I had heard of them; I'm pretty sure I know who that is. So Whby is one that I'm definitely familiar with; it was like a short-form uh Media Company. Yeah, yeah, and then there's British Volt, which looks like an electric company that went under. So, you know, some big companies I went under um in 2023, 2020, 2022. So that's interesting. So we have a lot of companies here, and we're just looking at um that had total laid off, but let's take a look; let's let's use Group by real quick. I want to look at the company, and I also want to look at the sum of the total laid off, and for that we need to use a group by the company, and let's just start with this, and I'm sure we'll use it order by in a second. Yeah, let's order by order by; let's just do two for now in descending, and two stands for one two; this is the total laid off, so uh for the total for this table, and we don't know how far go back it goes; we haven't checked that yet; we'll check that in a second, but for this table, you should recognize a lot of these companies. So I think it starts in like 2020 until like sometime in 2023, but this is Amazon; laid off 1,800 people; Google 12,000; I'm guessing that's at one time because that was the max that we looked at earlier; uh, this is Facebook or Meta; Salesforce; Microsoft; Phillips; Uber; Dell; Cisco; Peloton; I mean, these are a ton of big companies; Carvana; they let go of thousands and thousands and thousands of people; Twitter; that's not surprising, uh, given what's the change of things; Groupon; um, ton of ton of people or ton of companies, and that's a lot of people that have been let go. Now let's really quickly uh before we keep going, I want to look at our date ranges real quick, so let's select everything, um, whoops, we do from there, and how do we want to do this? Let's do minimum of date, and let me do it like this, date, and then we'll do uh the max as well, cuz I want to look at the date range that we have here. Let's run this; it looks like it starts in 2020 of 311, so right when like I believe the pandemic started or the uh COVID-19 started; I want to say that's like right when it hit at least us in the United States, then this is almost exactly three years later, so early 2023. So just in those three years, you know, here's some of what we're looking at; these companies have let go of quite a few people or had layoffs. We could also take this exact thing; oops, what did I do here? Copy this again; we can also take this exact thing and look at quite a few other things; there was um the industry, so we can look at industry like what industry got hit the most during this time or had the most layoffs; um, all we're looking at right now is total laid off; we can also look at um percentage in a little bit, but it looks like consumer got hit really hard; retail really hard; that makes a lot of sense with shops closing down cuz people couldn't come in for the coronavirus. Now we're just making assumptions, right, um, but you know, during that time, it was mostly COVID that impacted a lot of stuff. We have transportation, finance, healthcare, food, real estate.

Um, yeah, there's a lot, a lot of people. Let's look at the lowest ones: manufacturing, ftech, Aerospace, energy, legal. So low numbers on those. High numbers on these. So really, really interesting. Uh, let's go back up. Just want to look at our whole table really quickly, see what we got while we're looking at this stuff. And let's run this. Now we looked at the company; we looked at the industry. I would really be interested to look at the country as well. Which countries, at least from this data set? And we can copy, or we can go right here, country, because I believe that United States had the most. Holy mackerel, they had by far the most. Uh, then India; this is 256,000 people um, lost their jobs. We'll look—I think we'll look at the dates in a little while, like a kind of like time series—like how many per year, per month, per day, or whatever you want to look at. But goodness gracious, uh, that's a lot of people within just three years in the United States. India, Netherlands, Sweden, Brazil, Germany, United Kingdom, then it goes down and down and down. But these are just reported um, from this data set that I, I had gotten. So really, really interesting. Good night. The United States had much more than, than most, for sure. Um, let's actually look at that date real quick, or we can look at it by year. Um, so we have this date, and if we do it like this, and we can by date real quick. So this is going to do it by individual date, and let's order by—let's do one. So this is the most recent date, so it's literally by date that's reported. We don't want that. Let's do it by the year, year. So 2020, 2021, 2022, 2023. We can do that fairly easily. We'll use this year function, and we'll Group by the year as well. Let's try running this. There we go. It looks like in 2020, 880,000 people; 2021, uh, 16,000, 160,000; in 2022, this looks like the worst year; and then it's only—we only have three, three months of data in 2023. There's 125,000. Holy smokes. So in 2023, it looks like we're ramping up, because I'm recording this in 2023, about a month after this data set, uh, that we got this data set. There's 125,000 people um, around the world, you know, but just in those first three months. So this is going to be a lot higher than even 2022. That's pretty wild. Um, very, very interesting. One other one, one while we're looking at Group by um, there's, there was a column, and you can go back and look at it if you'd like, but it's called stage, and this shows the stage of the company. And if we run this, and we're all just looking at total a off, but if you look at the um, stage of the company, this is like the different series that they're in: A, B, C, D. A, I believe, is like a series A funding; that's like a super, super starting—oh, this is like a seed phase, then there's series A, and then it goes up, up, up, up, until usually they go um, like they do IPO, or they get acquired or something. Now if we go up here and we do two descending, I want to see which one had the most. So this is post IPO; this is the Amazons, the Googles of the world, the large, large companies that are post IPO or initial public offering. Then there's unknown; we don't know which that is. Um, a lot of, you know, layoffs from Acquisitions, C, D, B, all the way down. So it looks like um, most of it's coming from, you know, these ones right here. Really, really interesting. Let's go look at percentages. I'm just going to literally—co—trying to say literally—I'm going to literally copy these um, and with percentages, I don't think—uh, let me look at percentage—I don't think the sum is going to be a good indicator. I don't know if this is a good one to even look at, because—and then we're looking at company right now, because percentages refer to a percent of the company, right? So we don't have hard numbers, because we don't know how large these companies are. So now that we're actually looking at this, this percentage laid off isn't super relevant. Um, really the one that's kind of more, you know, has better—this is a better use for what we're looking at is this total laid off, because again, we don't know these sums. We could, we could look at like the average, right? Um, but again, that just doesn't help us that much, I don't think. Um, think we're going to really dive into that too much is my, uh, is my feeling. Now one thing that I would be really interested in is to kind of look at the progression of layoff, right? Uh, you could call this a rolling sum, so start at the very earliest of layoffs and do a rolling sum until the very end of these layoffs. Um, and let's go to the bottom. This is where it's going to start getting a little tougher, um, and there's, you know, we're just doing a little bit of exploratory data analysis, you know, do digging into this a little bit. You can go and dig into this as much as you'd like; you don't have to just do what I'm doing, but I'm just trying to show you some stuff. Now let's try to do rolling total of layoffs. Um, we could do that on the day, although I feel like that's going to be way too many rows. Let's do it based off the month, so right here in this month. Now let's see if we do just the month. Let's do something—I'll show you the month, and that's going to be an issue, and I'll—in my head I already know—but let's look at it. We could do something like select um, from—let's get this—there we go. So if we do um, we'll do substring—let me—that a sub—let's do substring, and we want to pull out this month right here, so we'll go one, two, three, four, five, six. So start at position six, um, and this is of course in the date column. We'll start at position six, and then we'll take two. Let's just run this really quickly, and there's our month. So this we could do as month, right? Um, or like—is that correct? Yeah, so as month. So this is our month that we're doing it. Now if we group on this and we do like something like a sum of total uh, laid off, I think that's the column, and then we do a group Buy on this month, so it'd be like this right here. We'll do group bu this. Let's try running this. We should be able to do month as well. Let's try this real quick as well, because I don't want to have this if I don't have to. Run it. Perfect. So the months right here don't show us the year. So if we're trying to get a rolling to total of just the month, it's actually could work fine when we actually implement the, the rolling total use um, you know, a window function, but the issue with this is it's just going to show us month. So this is 20, 20; this is January of 2020, 2021, 2022, 2023; any other years we have it. This is not a great rolling total. What if we did one all the way to—I want to say it's seven, six, seven. Let's try this. Now this is going to give us a much better—let's order this—order by—a one—sending. This is just our first column. So now uh, well we should do it where it's—not—give me a second—I'm, I'm, I'm figuring this out as we go. We'll do where uh, the month—write that—where the month is not null. I'm just going to get rid of that one, and of course uh, that doesn't work because we're looking at the substring. So let's try doing this. Let's—there we go. Um, it just wasn't reading in that month that I was trying to use. Let's go down. Now here's what we're going to do is we want to take it from the very first month, and we're grouping everything, so these are all the layoffs from 2020 of 03, so that's March of 2020, and we have April, May, and these are the layoffs. So this is really good; this is exactly what I was imagining in my head. So we want this—this is just, you know, 12 months in a year, and we go all the way to the bottom, and I want to do a rolling sum of this. So let's see how we can do that, and we'll use this logic in a little bit. Let's copy this, and let's do select everything. We'll do right here. Now what we actually want to do now that I'm thinking about is we want to take this data and we want to do the rolling sum based off this exact thing. So we actually need to take uh, this—let's get rid of this—and we'll do it with a CTE, so we'll say with, and we'll do rolling undor total that we'll say as, and then we'll put this in here just like that. So with rolling total as, now we're going to say select, and we'll just do from here. Now what we need to do is we need to select the month, so let's go ahead and select that month, and we'll take it just like this. So we'll select the month, and we need to do a rolling total. All we have to do for that is the sum of which column we're doing. Let's actually change this real quick. Um, we're going to call this as um, total off. I'm just going to keep it simple. So the sum of total off. So now we're doing that, but we want to do it over, and all we need to add into here is an order by. We're not going to Partition by anything, because in here we already did a group by, so it's, you know, kind of like partitioning it. We just need to say order by—we just need to order by the month, I believe. So let's try that, and let's run it. Let's do that, and we actually need to—since we're doing um, this—we need this at the end, and we can rename this if we'd like. So we can do this as rolling uncore total lowercase. Let's try running this, and let's see what we get. Okay, and this looks correct. So starting in 2020 of 03, we had 9,000 layoffs. Then the next total we added onto here—now this visually isn't the best. I would like the month right here as well. So let me actually add um, let me create its own row, put a comma here, then right here I want to keep this total off, so we can visually see better. Much better. Okay, so we have the month, and as it goes down, we're having more laid off. Now this is our rolling total. Here's essentially how this works: it starts with 9,628, then it adds on the next month, which is 26,000, which equals 36,000, then it adds on the next month, and we get 62, adds on the next month, 69, right? It keeps going all the way down. This just shows each month how many were laid off, and this shows a month-by-month progression all the way down to the bottom. So let's keep—let's just, you know, take a look. In 2020 of 03, we had 9,000. By the end of 2020, we had about 81,000 or so. Then at the beginning right here, all the way down to 2, 21. By the end of 2021, we only had 96,000. So 2021 was a good year, it looks like, um, comparatively. We had 90, 80—well, let me see—91,000 people let go, and here we only have 96,000 let go, so that's what our 81—that's only like 15,000 people; that's like nothing um, comparatively. Then in 2022, uh, things start ramping up dramatically. It looks like we have um, 12,000 people, 17,000, 16,000, and they're adding up; it's going from 97 all the way up to—good night—right before the holidays in 2022 of this past year, I mean, we had uh, 247,000 people. So that's like 100 and 30 some thousand—my math—my math's really bad—it's like 150,000, and then we only have—oh, we have even more here actually—and then we only have the first three months of 2023. So these months right here were really devastating, just around the world. Now we can also break this out potentially by country, so we can see how many per country, but this is just around the world. That's a lot of people losing their jobs, all the way up to 383,000. So in this range, 383,000 from March of 2023 all the way back to March of 2020 lost their jobs, and this is just reported. I'm sure there was uh, you know, much more than that. This is that—like companies, larger companies that have like series A funding, IPOs, etc. Um, but a lot of small businesses went out of business, um, so we don't, we don't have that information in this data set. So I think that's what we're going to do next is kind of look at the company maybe, because I'm always interested in the company. And actually earlier—let's not do that one—earlier we're looking at the company, the sum of total a off. Let's um, let's bring this down. Let's run that CU. That's what rolling total is, by the way. Rolling totals are great; I'm really good for visualizations as well. Um, let's see. Yeah, so I want to take a look at these companies, but I want to see how much they were laying off per year. So instead of just looking at it as a total, we'll break it out by the year. Now I'm just going to warn you, this probably going to—this most likely be our last one in the, in the lesson. This is going to be probably our hardest one yet, um, potentially. We'll see—maybe the other one was earlier uh, was harder earlier. Now let's use this kind of as a starting point, but what we're going to need to do is we want to take the company, but I also want the date, so I need to do a comma, then date. So we need our date here, and I'm going to do that. I need to groupy the date as well, so we'll do date, and let's run this. All right. Now this is just doing the, you know, company and the exact date. We don't want to do that. Let's actually do the year. Let's just look at the year; I think that'll be plenty. You can also do the exact same thing as we did above with the, the substring, um, although I think that's going to get a little Messier um, off, you know, just a thought. Let's run this. Okay, so now we're looking at just the Year; we're grouping by year. Let's order by uh, let's say the company, and we'll do that in sending. There we go, and let's run this. So now we have it open. Let's see who—you know—you can see people who made multiple layoffs. This is in 2020; they let go of 200, and then in 2023 they let go of 100, 55. This a company I've never heard of. So this is already looking really good. Now let's say we wanted to use this, and what we want to do is we want to rank which years they laid off the most employees. Now this is just a small uh, sample. We'll look at more in just a little bit. We can actually look at um, let's just do three uh, three descending just like this. Should be large companies. So, you know, some of these companies like Microsoft, even Amazon right here, and Amazon right there, they let go of multiple or thousands of people in different years. So I want to rank those. I want to say, you know, the highest one uh, based off of the laid off should be ranked number one; that's the year that they laid off the most people. So let's go ahead and try to do that. Thing we need to do is uh, do a CTE; we'll start with that. Let me um, me add some more things down here so we're good to go. So let's do—we'll do with—let's do uh, company—so this is going to be the company year underscore year—we'll do it as—and that's what this is going to be. This is our company year, and we can do select everything from company year. It's going to be the exact quer that we're looking at. Let's go ahead and run this. Okay, so this is good. Now I do want to change these columns, and I can do that right here. We'll do company um, let's call this years, and then we'll do—I'm going to do total laid off again—so total undor laidor off—this the sum, right—total laid off per year. So let's go ahead and run this. Now there we go. We have company, years, and total laid off. So this looks much better. And what we're going to do is Select everything, but we want to partition it uh, probably based off this years right here, and then we want to rank it based off how many they laid off in that year. So we'll get to see who laid off the most people per year, cuz some companies like Amazon, they let—they let off multiple people per year. What was it, the highest per year? That's kind of what we're going to look at. Um, so we'll do Dore Rank, and we're going to do that over. Now we're going to Partition by—oops—that's not how you spell partition—Partition by—we want to Partition by the years, so all of the 2021 layoffs will be in the same partition, all the 2022 will be in the same partition, and we'll do year, and we want to also order by the total laid off. Now we want to do that in descending, so we'll do total laid off descending, and then we want to um, add this dense rank to it. So let's try it. Let's run this. Good night. That's a, a big one. So let's take a look. So in 2021, it looks like um, or 2020, it looks like Uber had the highest, and we want to take out the NES. So let's do um, where years let's say is not no, and let's run that. Here we go. So in 2020, and that's what we're partitioning on first, it looks like this is one, two, three; these are the top ones, um, and let's order by, and let's do the—let's order by the rank um, first. Let's call this as—bring it down—do we want to call this—we'll call this as ranking. There we go. Order by ranking ascending. There we go. Now we have our ranking. So in 2020, this is the biggest one of layoffs; 2021, this is the biggest layoff. I guess we'll have to take a look in Meta. In 2022, they had the biggest layoff, and Google had the biggest layoff total uh, for 23. So this looks correct, but I kind of want to filter on this ranking to be able to only filter maybe the top like five um, companies per year, and I think we can do that. Let's actually get rid of this. I think what we should do is we should add this as another CTE and query off of that. So now we'll call this company underscore year underscore rank. So now we have the year rank as—we'll have our query—oops—have our query. So now this is our company your rank. So now if we do select everything from company your rank, this—we run it. Okay, so now we have our rankings. Let's come down. Now we have our rankings, but I just want to filter it based off of that ranking. We'll say uh, where ranking is greater than or equal to—Let's say five—we'll look at the top five rankings. Let's run this. And I say greater than—I wanted uh, less than—that that's looking good. Okay, so really quickly, we have in 2020, we had—these are the top five people who laid people off: Uber, booking.com, Groupon, swiy, Airbnb. In 2021, the largest layoff was ByteDance, which I think is TikTok, right? Catara, Zillow, uh, yeah, these are top five. So 2021 and or 2022 and 2023 were definitely the largest as well. We have Meta, 11,000 people; Amazon, Cisco, Peloton, and Carvana, as well as Phillips; they tied, that's why we have the dun ranking, because some of these will be ties. Then we have Google uh, in 2023, all the way down to Dell. These are all ones I know: Microsoft, Ericsson, Amazon, Salesforce, and Dell. So this is really, really interesting, just looking at a year-by-year snapshot, right? These are the total laid off for each company, and we could even go back and change this for like industry or, you know, really whatever we want to change this to. This is just an interesting query in general to look at, you know, per year, and we could go back and change per month, or lots of stuff we can change in here, but this is really interesting to me. Um, it just looks like a lot of the large tech companies had some—took

Some big L's took some big hits. Um, let's recap this query really quickly, in case you know, it's tough to follow. But we created this query up here, and we were looking at the company by the year and how many people they let off. Then, right over here, we said with the company year, we changed these columns; this is our CTE. So we created our first CTE. Then we went and we gave it a rank, and we wanted to, you know, filter on that rank, so we did this rank as another CTE. We just did a comma, had a second CTE, and we hit off the first CTE, the company year, which is right here. So we hit off our first CTE to make the second CTE. And then finally, we um queried off of the final CTE. Definitely not an easy query to kind of think through and walk through, but I hope you know you're able to follow, um, because you know that's a a really good query. This is something I've definitely done in a real job when I was working with a lot of healthcare data. This is a a lot of stuff that I would do, and so this is a, you know, pretty good, um, pretty good query to know how to do.

But with that being said, uh, we are done with this lesson. I hope this wasn't too short. I don't know how long I ran, but um, you know, we looked at a lot of different stuff. Let's go back to the top again. We were just exploring the data; we looked at laid off a lot, um, looked a lot at the company, uh, when these dates actually started for these layoffs in this data set. We looked at the country, the actual year of laid off, uh, then we went to a little bit more difficult things; we looked at it per month, so per month how many layoffs they had, and then we did a rolling total. This one was a pretty good one using that substring. Um, I love substrings, man, they're awesome; or lady, they're awesome. Uh, and then we came down here, and we did the one we just did with multiple CTEs in the company. I think it was a a really really good solid project. Um, combine that with that data cleaning project, and man, you got just a really good start with some MySQL projects. And this one can be expanded upon. Don't stop where I stopped, right? Let me go back up the top. Don't stop where I stopped, right? This data set has so much data in it; you can do a lot of different things. And even if you want to, you could go and find these companies right over here, and you could try to uh get their total, uh, total company that they had, and you could use this column a lot more; that'd be really interesting with some calculations there.

So with that being said, that is the end of our exploratory data analysis project. I hope you enjoyed it. I hope you learned something, both in the data cleaning project and this exploratory data analysis project. That's what this is all about: and getting the confidence and gaining the experience to create these projects and add those to your portfolios. Speaking of which, if you haven't already, check out my video on how to create a free portfolio website, uh, using GitHub. Awesome. I highly recommend it; you can add these to your portfolio. So with that being said, thank you, thank you so much for watching. I really appreciate it. If you like this video, if you learned anything at all, be sure to like and subscribe below. Check out my channel for tons of other videos just like this one and more. I will see you in the next [Music] video.

What's going on, everybody? Today we are starting our Excel tutorial series [Music]. Now there are so many things that you can do in Excel, so I don't know how long the series is going to be; it could be 15 or even 20 videos. But what I do know is that I'm going to be covering just about every single thing that I've used since I became a data analyst, and I want to show you how to do it, uh, so it won't just be the more concrete things, um, you know, like pivot tables, charts, V lookups, things like that; it'll also be some of the more nuanced things, like how to deal with missing data or how to deal with dirty data and how to clean that up within Excel. And so those are things that you may not be able to do, you know, if somebody wasn't showing you how to do it. And so that's what I'm going to try to help you, because I know that that is something that you will need to do or learn how to do in Excel.

Now, before we get into it, I want to give a huge shout out to the sponsor of this Excel series, and that is Udemy. I took so many Excel courses on Udemy when I was first starting out as a data analyst, and there was this one course that I kept going back to over and over again, because as I got into it in my job, I realized that there were so many things that were in that course that I really needed to know, but I didn't realize I needed to know it. And so I'm going to put the links to those courses in the description in case you want to take those. Again, huge shout out to Udemy. Without further ado, let's jump onto my screen and get started with our very first Excel tutorial. All right, so I'm going to go ahead and get rid of myself. We are going to be looking at something absolutely pivotal in your data analytics career, and that is Pivot tables, uh, and I think that's really appropriate. It is probably one of the most commonly used things I think that data analysts use to convey information in Excel. It's super easy; easy to group things together, to display information in a very easily understandable way, especially for people who are not data analysts, right? I use this a lot for other managers or for higher-ups, um, who don't want to get into SQL or or you know aren't super text in like Python or Tableau; they just want it in in Excel. And so I use it all the time for that reason. And so we're going to be using this data set right here, Bike Store Sales in Europe. I will include this link in the description. Um, we're not going to look at the columns just yet; we're going to download it. Um, I've already downloaded it a few times, but we are going to go to um our downloads; we're going to open it up, and we're going to open up this sales right here and give it a second. All right, perfect. And so here's what it looks like, at least on my screen. I'm going to uh spread it out just a little bit, um, and really quickly let's take a very quick glance at this. So we have a date, a day, a month, a year; years; so some, um, some date information, um, then we have some customer age information, so how old was the customer? Again, this is bike sales, so what did um, you know, what did they buy? And they have some demographic information, so this is their age group. We have uh the gender, country, State, uh, the product category, the subcategory, the actual product that was purchased, and then we have things like um, you know, how much the these things cost, the quantity that was that was ordered. So we have order quantity, unit cost, unit price, then we have the profit, cost, and revenue; all things that we almost everything in here we can in some way put into a pivot table.

Now I'm not going to go through every single variation of that, but we are going to be um looking at a lot of this um Revenue over here, because I think it's it's pretty easy to show the value of a pivot table with, especially with um, you know, current your money. So what we're going to do to get started is we're going to go up to insert, and we're going to click on insert, and then we are going to click on pivot table. Now really quick, there is a recommended pivot tables, and if you click on that, what will come up is some recommendations that Excel gives based on the data that you have, um, and it can kind of give you some ideas of of what you can do with pivot tables; it's going to generate it for you. We're not going to do that; we're going to build our own, uh, but let's on pivot table, and it's going to Auto Select basically everything, and that's fantastic, um, but what if it doesn't come like that? I I just erase that; if it doesn't come like that, you can click right here; you can kick, excuse me, you can click control shift and then the right arrow and then the down arrow, and that is going to select all of our data, um, and you have right here a new worksheet or an existing worksheet. We're going to create a new worksheet; just tends to get too clogged up if we put it on the same worksheet that already has a lot of data in it. So right over here are pivot table fields, and these are all of our columns that we just looked at, and we're going to be able to select those and kind of drag and drop. Now if you just took the Tableau um tutorial series that I just finished doing last week, then this is going to be pretty familiar; you're going to start seeing a little bit of um, hopefully some patterns about how the data is kind of displayed. And so we have our filters down here; we have columns, rows, values, all these things, uh, we will be using; I'll show you how to use today, as well as some additional things. Um, one thing that we want to start with, uh, for this demonstration is we're going to be looking at kind of the um these bottom ones right here, profit, cost, and revenue, and we're going to be doing that per country, uh, per country and state, and we'll kind of do some drill Downs, um, and I'll show you how those work. So for just to start out, we're going to take the country right here, here, and you'll see it populate right over here. In fact, um, let me zoom in maybe once, uh, yeah, that should be fine; I don't know if I want, I might zoom in it again in just a little bit, um, so we have our country and and it's just like this very very simple oops, um, now I'm going to include the state. Now I'm going to drag this um all the way, and I'm going to put it under; you can put it above or you can put it below; I'm going to put it below, uh, it definitely makes the most sense there. Now when you do that, it it um kind of populates it in an expanded way, but you can collapse this very easily. We're going to go right here; we're going to right click; we're going to go go down to expand and collapse, and we're going to collapse the entire field. And so now here are all of our um all of our countries as they were before, but now each of them has this plus sign to the left, and if you click on it, now we can go and we see this state that we that we added to these rows. And what this is going to do is it kind of is like a rollup or it's like a grouping, um, and so if you you know have taken the SQL um tutorial series and you've done uh things with Group by, this is very similar to that, um, and if you've done the Tableau tutorial series, it's kind of like a drill down; it's very very similar. So you can drill into the information. So we um can put some values in here, uh, and what we're what that's going to do is that's going to kind of create some some context to what this will we're grouping by. So just for um visual purposes, let's add this Revenue. So this is the revenue that is bike uh bike sales revenue, right? That's what we're looking at. So this is the sum of the revenue for these bike sales per country. Now if we drop down right here, we can see that in Australia, uh, New South Wales had uh 92 what is that 9,23,45; Queensland had 5 million, you know, etc, etc. So now we can break it down; we can't it's we don't just have to look at Australia; we can now drill down even further to the actual state, is what they're calling it, um, the actual state within Australia. And so it's super super useful, and you can do that for every single one. And so we can look at Canada; we can look at France, and we can really drill down into uh the revenue for each of these countries, as well as the states within them. Now over here, this is not the most uh pretty, um, it just says sum of Revenue, and then it has some numbers; not not the most pretty thing I've ever seen. Um, really quick, we can go like we can um kind of highlight over these, and we can go back to home; you can do it in a couple different ways; we can go to home, and we'll type currency. Now it has these two .Z at the end; you can get rid of those really easily by going like that, um, already this looks quite a bit better, just visually, um, especially if you're looking at it in uh, you know, dollars; you can change the currency um to different currencies if you want to do that.

Now we don't just have to do uh the sum of Revenue; we can do a lot of different things. So let's go to the value field settings. So we can customize this name; so we can do um Revenue oops, you good if I get spell Revenue per country; that's f fine, that you know it's just a placeholder, trying to show you, but we don't have to just do that, um, you know, we could do the count, the average, the max, the min; we can do just about anything we want, um, but let's keep it the sum right now, um, and if we want to, we can show this value as different things; so we percentage the uh percentage of column total, percentage of row total; let's do really quick just for demonstration purposes the percentage of grand total. So when we do that, we can see that the United States, the per Revenue per country, United States has 32%, just between these um, you know, these countries, and Australia has the next one. So you know it might be kind of hard to glance at this really quickly to know who has the highest, um, but what we can do is we can go right here, and we can go to sort, and we can do largest to smallest, and there we have the United States on top. Now when you do it right here, it's not sorted largest uh to smallest; you'd have to go and again click sort and do largest to smallest. And so now we can see that California has the has the um, you know, biggest percentage; they're pulling in 20% of that 32% of Revenue. So I'm just going to click ctrl z a few times and get us back to where we just were, um, and what I want to do is I want to show you a few different things, uh, pretty quickly. So we want to pull in this profit and this cost, uh, and so I'm going to pull in this cost next, and then I'm going to pull in this profit again, uh, I'm going to change the currency on this, and I'm not going to change the names, um, right now, but you you absolutely can do that. Now the revenue is the how much is actually being sold, so you know for the United States it was 27 million. Now the cost is how much did it cost to manufacture or or store, um, or or distribute all of these products, so that was 60 million, and the profit is actually how much money is being made at the end of the day, after um, you know, all their costs, after all their employee cost, after everything, they're still making the United States is still making $1 million. Now you might look at this and you might say, well, you know, I can kind of glance at it and say know that this profit is correct based off these two numbers, um, but we can do a calculated field, um, and if you remember what calculated fields are, that's something from Tableau; very uh basically the exact same thing. And so we can create an additional column right here that is a calculated field that can add and subtract these things to make sure that our numbers are adding up correctly. So let's do that really quickly. U let's go to pivot table analy; we're going to go over to Fields, items, and sets, and go to calculated field. Now we can name this anything, um, and I'm just going to for demo purposes I'm going to say um oops calculated field demo, uh, I'm sure yours will be different. Now, um, if you want to, you can go in here, and this is the formula; it's almost like um, you know, we haven't looked at formulas; this is our first tutorial, but you know when we look at formulas, it's basically the same thing as writing it if inside of a cell, but here it gives us kind of this um open text to do how we uh do what we want with it. Now what we're going to do is we're going to do Revenue; I'm going to insert that, uh, I'm going to get rid of this; I'm going to do revenue, and so that's the the the very large number, and then we're going to subtract, and we're going to subtract our cost; we're going to insert that, and let's do this and click okay. So this is our calculated field demo column that we just created, and as you can see, it matches our sum of profit column exactly, and that's exactly what we want to see; we want to kind of check to make sure that this revenue and cost uh fields are generating the correct profit, and sometimes those are off, and so it's really good to kind of check those and have that additional column. Um, You probably wouldn't have this if you were um, you know, going to submit this to somebody, uh, just so you know. Now that this is an actual column, you can't go here and do something like cut or and paste it over here; you know, that's not it won't let you do that; what it is is is now an actual um column. So we can go and remove that, and we can add it back at any moment. So if we want to go back and add that um oops add that down here, we can do that because we've created that column; it's now permanently there unless we go and delete all of that data, uh, and so we can just click this check mark, and it will get rid of it for us. All right. Now the last thing that we have not used down here is the filters. Now the filters is exactly what it sounds like; it's going to allow you to filter on certain things, um, but probably not things that you already have included in your pivot table. So if you add something like the country down here, um, it's going to kind of expand everything, and then if you then go and filter on it, it kind of breaks it down; that's really not what the filter is kind of used for or meant for. Um, for example, right up here we have customer gender. Okay, so let's take the customer gender and we'll put it in this filters. Now we can see all of the revenue, all of the cost, all the profit, and we can do that based off of the gender. So we can filter by a gender, not really having to change anything about our pivot table. And so at a super quick glance, we can see that uh the males are the profit from the males is 16.48% is 15.73 3 million. So at a super uh basic level, at a really quick glance, we can see that the men or the males are you know spending a little bit more than the females by about about $700,000.

Now let's go ahead and create one more pivot table, uh, we're going to create a pivot table right over here. Let's go back to the sales right here again; control shift right down; it's going to select all of our data, and we're click okay. So one thing that we're going to look at is we're going to use some of this date uh information right here. So let's select our country just like we did before, um, and what we want to do is see, you know, what year were we performing our best? When were we doing our absolute best, uh, with oops, let me go back, uh, with our sales. So I'm going to select the year and put that in our columns, and

So now we have 2011 through 2016, and we want to look at our Revenue. Let's put our Revenue right down here. And now we have all of our Revenue. Now let's again make this into a currency, just like that. And super quickly now we can get a really quick glance at how Australia was doing each year, and we can see that there was a huge uptick in uh 2013 and a huge uptick in 2015. That didn't happen for every single country. Uh, did go up uh for most countries, very slightly for some, but we can see on a large scale from um year to year what that's like.

So within just a few minutes, we're able to create some really useful pivot tables that anybody could look at and understand. And that's really the biggest use of these pivot tables is that you can kind of group these things together, show some uh information, data at at kind of a broad, larger scale and make it to where anybody who's looking at it can understand it. That is why pivot tables are so useful.

And so I hope that this video was helpful. I hope that I was able to walk through it and help you better understand how pivot tables work and how you can use them when you are working within Excel. Thank you guys so much for watching. I really appreciate it. If you like this video, be sure to like and subscribe below, and I'll see you in the next video. [Music]

What's going on everybody? Today we're going to be looking at formulas in [Music] Excel. Now I know what you're thinking: there's absolutely no way that you're going to be able to show us every single formula in Excel. And you're absolutely right, but I am going to show you some of my favorites and the ones that I found the most useful. And then you can go ahead and practice those and try those out. And if there are ones that you really want me to do and you think that I missed, put it in the comments below, and I will see those, and I'll try to make a list of those and make another video on formulas and include all of those as well.

And now before we jump into the actual tutorial, I want to give a huge shout out to the sponsor of the series, and that is Udemy. You guys already know, if you have watched any of my videos, that I absolutely love Udemy. I mean, honestly, they were the ones who got me started and were able to give me affordable courses for me to get started as a data analyst. I learned SQL and Excel and Python all through Udemy courses. And so if you were looking for a platform to take a course, I absolutely recommend you look at Udemy. They have fantastic sales going on right now, especially during the holiday season and this new year. And so if you're looking to take a full-fledged Excel course, I have some of my favorites in the description below. And now without further ado, let's jump onto my screen and get started with the tutorial.

All right. Now before we start, I want to say that this is not like every other tutorial that I have created. This one is very streamlined, okay? So I already know exactly what I'm going to do; there's not going to be much messing around. I've lost little notes here and there. Um, you I'm going to try to get through it because there's a lot of them to get through. Um, so all these ones at the bottom, now these are ones that I use a lot that I think are useful. Again, if you know other ones that you use a lot that think that I should be using, which I know there are ones that I left out of here, you know, put it in the comments. Um, I'll see the ones that people are liking, and I will I will create more videos on these because I know there are so many. I also will save this um Excel in uh on the GitHub so you can go and download it. It'll be exactly what you're looking at right now. I highly recommend trying these formulas out for yourself so you can get a feel for how they work and how they're actually used, and you can mess around with it yourself. So um as you can see at the bottom, we're going to start with Max, Min, and then we're going to go on to some more I think a little bit more uh difficult things, um and all these things are super useful. I'll try to talk about how you can actually use it as we go through it. Some are super self-explanatory, but some may not be.

So this one I think is super self-explanatory, but again, one that you're going to use all all the time. Um, and so what we can do is we can say equal, and that's how you kind of start off saying this is going to be a formula in this cell. Equal means uh I am now creating a formula, and we're going to say MX, and I'll hit Tab, and so it'll kind of populate it. And right here, if you've never seen a formula before, it'll kind of give you what the inputs need to be. So it's going to say Max of number one, number two, etc., etc. What we're going to do is we're going to give a range, so we're going to go from here down to here. You don't have to close the parenthesis, but you can. I'm going to, and then you hit Enter. And so for this date, it's going to give us the max date. Now these are um the start dates for these people right here, and so if we just kind of glance through here, we can see that 2013 was the last year, and this one is actually the latest in that year, and so it gave us the correct one. The Min is going to do the exact opposite; it's going to give us the smallest. And so we'll give it the same range, we'll close a parenthesis, and it's going to say December 7th of 1995, and we can see that that is correct. So Michael Scott started in 1995, the earliest of all the employees. Um, and you can do the exact same thing for really any of these columns. Um, we can see who the who's making the most money or at least what the highest salary is. U So we'll do Max, and then we'll do the salary range. And so this is this one again uh whoops, what I do? Oh, I did the wrong range, didn't I? No, I didn't do the wrong range; it's just there it goes uh this column was a date range or a date column for whatever reason. Let me get rid of that, uh and then we can do equals Min, and we'll do again we'll do the salary. And at a quick glance, we can see that Pam Beasley is making the least, and 65,000 is Michael Scott who's making uh that. So super simple; it shows the max, it shows the Min; you can select a range. There you go. Let's move on to IF and IFS.

Now IF is um I think pretty straightforward. So all you're going to do is you're going to say if this, then that. Um, IFS is a little bit different. So IFS is you can you can put multiple conditions, and as we're writing it, I'll show you kind of what it the conditions that need to be met. All right, so we're going to click right here; we're going to say equal; we're gonna do IF, hit Tab, and we need a logical test, uh and so we're going to give it a range or or or something. We're going to say if it's equal, greater to um something like that, then we're going to say if the value is true, what's the what is going to be the output, or if the value is false, what's going to be the output. So let's do this right here; we'll do this age range. And so if they are greater than let's say let's do 30, if they're greater than 30, we're going to do a comma, and so if the value is true, what what should be the output? Uh, if they're greater than 30, we're going to call them old, and then if it is false, so if they're younger than 30, what should it say? And we're going to say young, and we'll close the parenthesis, and there you go. So if they're over 30, then they are going to have or if they're younger than 30, they're going to have young. Now this is something where you need to specify if you want 30 over or over 30. We chose over 30, so 30 is not included in that, um so they're going to be young. Now uh let's get we don't actually need two of these; that's pretty self-explanatory. The IFS is a little bit different, right? You can have multiple conditions. So let's open that up real quick. So IFS, and now we have a logical test, a value if uh that's true, then you can do logical test two, Value if that's true, um so you can have multiple multiple multiple things. Now this one is a little bit different; in this one oops, let me get out of this; in this one you had a value of true, a value of false. IFS does not have that. IFS is going to give you um different ranges and different specific conditions, and you can't say if this one's false. You're just going to have multiple conditions. So let's do equals and IFS, Tab, and we'll do our first logical test. So let's do um if the salesman or if that equals to salesman, we're going to say we're going to respond with sales. So that's if the value is true; that's what we want the output to be. Now we're going to go on to our logical test two. So you're going to see this pattern right: if this is our conditional or logical test, so if this is true, this is what's going to be returned. So you'll notice that it's just a a pretty simple pattern. We can just do random things. So if it's equal to sales um and we'll just do the same one, if that is equal to say HR, we can say fire immediately. And now we're going to say if it's equal to regional manager, and we say give Christmas bonus, and we'll close the parenthesis, and let's see what we get. So as you can see, there's no default value for true or false like like this one; there was a logical test, and if it was true, there was a value, and if it was false, there was a value. So for every single one, you'll get a value. For this one, that's not exactly going to happen. As you can see, there are these NAs. Now when that happens, it just means nothing met that condition, so we never said anything about supplier relations, we never said anything about accountants, but if it was part of that IFS statement, then it got something. Um, and so that is how the IFS works. Now let's move on to LEN. Uh, this is exactly what we're going to do, but you know some of the uses for this U for the LEN. I've used it for a lot of different things. Um, one thing that I've used it for in the past, and you know Max and IFS, you know, you can use it for almost anything. LEN is there's a lot of different use cases. One I used to work with a lot of um customer data or patient data; they had like Social Security numbers, and if you know there was bad Social Security numbers, we didn't want to include that. And so we do like the length of that, and if a social security number was let's say 10 numbers or 11 numbers where it should only be nine or or you know however many they are, I think it's nine, then we know that that social security number is incorrect, and then we can get rid of that or discard it from our results. That's just an example, right? Um, so for this oops, what I do? That I did control Z to undo that. If you didn't know how to do that U so we're going to do equals LEN, which is length, um and again, if you didn't see that, it returns the number of characters in a text string. So let's go right here and let's go to uh let's go to their last name, and we'll give it a range. So it's going to tell us how many characters are in that string. So for Halpert, it's seven characters; for Flenderson, it's 10 characters, and we're able to see a length. And so again, there are a lot of different use cases for this uh the social security number was one; another one is phone numbers, right? If you look at the length of the phone numbers and there's uh ones that are like 12 numbers long, you know, those might not be ones that are accurate, and you need to go look at them and see if you want to include them in your results or your output. So that is how LEN is done. Let's move right over to LEFT and RIGHT. Um, I I might be going a little fast, but uh you know, I'm keeping it I'm keeping it live; I'm keeping this on our feet, uh so let's keep going. LEFT and RIGHT um are kind of like substrings. If you've taken the the SQL um tutorial series that I've done, uh substrings are where you can choose a certain part of the text string and you can extract data from that, um and usually have to reference a certain number, so a certain amount of characters. That's the exact same thing, except uh unfortunately there's no substring; there's SUBSTITUTE, but there's no substring. LEFT and RIGHT is really the closest thing that we have. So let's kind of take a look real quick and see what we can do. So we're going to do LEFT, and it's going to say return the specified number of characters from the start of a text string. So we're starting from the very far left, and we need to choose our text and then choose the number of characters that we're going to be looking over. So let's go over here and let's just choose, you know, start simple, uh we'll get a little bit more advanced. So we have um this is our text range, so these are the the the ones that we want to look at, and then how many characters do we want to look forward? Um, and we'll just choose three as an example. And so you can see that it takes the first three characters from every single um thing. Now you can also do this with numbers; it doesn't just have to be um you know, name with with actual words or letters. You can do the exact same thing. So you can say RIGHT, um and we're going to choose our our string, uh and let's do this one. So you know all of them start with 100, um and we'll just say we want to take the last one. So this one is going to start from the very far right and go over one character. So right here you can see this is our range, and I just chose one, so starting from the very far right, we go over one character, and that's what we take. And so that can definitely be useful. Another one that you can do, and this one is one that I have used so many times, I mean honestly countless times in in actually using this in my job, uh so we're going to go from the right, and we're going to look at a date. So you know sometimes you have these date structures: month month day day year year year or year um you know day month year, all these different, and sometimes you just want to extract either the month or the year or or something like that, the day. And so we want to come in here, and we're just going to extract the oops, I wanted to make that range; we want to extract the year of the start dates. So we're going to do that, and then we're going to go over four because we want to take the first four characters from the right to give us the entire year. So let's do that, and now we can see exactly the year. And this can be just super super useful. This is again one that I've used a lot, and so that is one that you might want to remember in case you're ever doing analysis on you know, start dates or or anything with um date data. Uh, again, one that I highly recommend for remembering. Let's go over to TEXT. I actually probably should have included that um before CU. I actually used it in this one. Um, if you notice right here, this is a text. So in in this one we just did that was a text. You can't do this right on um start and end dates when it's a date uh format. And let me show you. So this is a date. Now if I do equals and you know we just did this uh let's do on the end date and me I'll do the whole range, give me a second, and we'll do four; it's giving us completely random numbers. Why is that? Because underneath the date range, there are um numbers, right? So if I go right here and I make this a general, it's going to have a numbers, and look, these are the first four characters from the right, and so it's doing what it's supposed to do, but uh it's not doing what we actually want, and that's the issue. So how can we convert this? Now there are a ton of different ways, um but the quickest, probably the easiest besides actually writing writing it out like this, like 11-2-2001, which then converts it to a date format, um but what you can do, you know, just so you know, you can create it as a text; you can do 11-2-2001, and now it will stay a text string. And as you can tell, these are a little bit different because this one is uh formatted or situated on the right, and this one's on the left. That's how you can tell the difference. Now if you don't want to do it by hand uh completely manually and waste hours of your time, you can do it in a very simple way. So we're going to do uh TEXT. So this is the exact um formula that we're going to use. So let's get rid of that one ops, there we go. So we're going to do equals, we're going to do uh oops, TEXT. It says converts a value to text in a specific number format. So for a date format, we can choose a date format, and then it'll convert it to a text for us, which saves so much time, I promise you. Uh, let's do all of these just like we did, and then we need to tell it what the format is. If we don't if we tell it something incorrect, it's going to give us a completely terrible output or just give us an error altogether. So this is a DayDay month Monon year year year year format, and that is what we're going to do. So we're going to do dd mm yyyy, and close that up, and there you go. And now we well, because it's in a formula, what we need to do is copy this and paste it right over here, and now you can see that is a general. This is something that we can use as a string. And let's just check it just to make sure. So we're going to do RIGHT; we're going to do this one; let's do all of them, and we'll do four, and there you go. So now it works; that is what we are looking for, um and you can do that. Imagine doing that with millions of rows or you know, let's say 10,000 rows; it's going to be a breeze, right? It's going to take you two minutes or a minute to do everything that you want to do instead of having to just do a bunch of mess to convert it to a string, which I promise you I've done, and it just takes forever; it's it's terrible. So that is uh TEXT, super helpful formula. Let's go over to TRIM. Now I I purposefully messed up this column. Now why did I mess it up like this? Because when you're working with real data, you're going to get data like this; it it's messy, it's dirty; it just has random spaces at the end for no reason, um because sometimes you're going to be working with um data that is inputed by a User; it's not like a drop-down option. So imagine somebody's typing this in; they accidentally put a space, so they accidentally put an Enter or something, and then they submit it, and this is how it's going to look in the database, um and if you're a data engineer or you know you're working with the raw data, if they don't clean that up, then you're going to be working with that that dirty data. And I I guarantee you if you're working as a data analyst, you're going to see stuff like this, not with maybe a last name, but all sorts of data. So we're going to go right

Here we're going to say equals TRIM, do open parenthesis. Actually, this says removes all spaces from a text string except for a single space between words. So like, you know, if it said Halpert space uh or gy space Halpert, it won't take the space in between there because it it kind of understands that in normal language a space is supposed to be there, so it won't do that. Um, but we'll take that, we'll give it this Range, close that up, and there you go. Now it is nice and clean, much more usable.

Now let's look at CONCATENATE. One that I have used just way way way too many times. Um, and something that I've used CONCATENATE 4, and you'll see this one in a lot of demonstrations for a good reason, is because a lot of people use it for this. Um, so what you can do is you can say equals um, and well let me tell you what CONCATENATE does real quick. So what CONCATENATE does, oops, I'm totally messing up here, um, but it joins two or more text strings into one string. It basically joins things together and adds them together. So let's do CONCATENATE, and we're going to add this first and last name again, one that gets used all the time, but that's because um it really is useful. So you can do this, and you can say now I want to include this. So concatenating this and this, and let's take a look. So it says Jim Halpert U, but it's all connected, and that's typically not how people write their names. So what we can do is we can go back in here, and we can do what my demonstration up here already tells us to do, which is we're just going to add another thing in here, and if we add two parentheses, we can include anything in here. We can include a dash, we can include an exclamation point, or we can just include a space. So let's just include a space really quick, and just like that, it works perfectly. And so now we have the full name.

Now something that you could use it for is something like generate uh an email. This is something that you absolutely could do, um, and it's, you know, pretty simple. So I'm going to do it like this. I'm going to say oops, what did I do? I'm going to say um . and then at the end I'm going to say @, oops, comma quotation gmail.com, and now I've created emails for all of these people. So just something that you can do with this, um, and something that it it absolutely is used for, and you'll see that demonstration almost everywhere because honestly it gets used a lot um by data analysts. And so uh, you know, just a good one to know, understanding how that that concatenation works.

Let's go over to the next one. So we are going to do SUBSTITUTE. Now SUBSTITUTE's really interesting. Um, there are different ways you can do it. I'm going to show it to you on these dates real quick. Uh, that's what we're going to look at. So changing a date format, changing how what it's supposed to look like is absolutely something that happens all the time, and um, you know, sometimes you'll even get it like this where it'll look like it'll be messy, it'll be different a different um I guess format. So this one has all the other ones have um slashes where these ones have dashes, and you know what you can do is if you want to, well let me actually go with the no instances real quick because this one is uh actually makes the most sense. Um, so we'll do equals, and we're going to say SUBSTITUTE, and oops, and let me say SUBSTITUTE replaces existing text with new text in a text string. So if we do an open parentheses, it says we take the text, we have the old text, we have the new text, and then we have what instance or how many times or or or what instance are we looking at it, and I'll explain that in a little bit. So the text that we're going to be looking at is this one right here. So let's take this range, and the old is we're going to take this Dash, and so let's take the dash, and then what do we want to replace it with? We want to replace it with this slash right here. I think it's a forward slash, isn't that what it's called? It's called a forward slash. Am I crazy? Um, and we're not going to put an instance. So notice that that's in a bracket, that means it's optional. We're going to do none of that, um, and what it's going to do is it's going to fix this. So this one is now in the correct format that we want, uh, and that's fantastic. That's, you know, that's what we tried to accomplish given what we had.

Now let's fix that. If we want to do the exact same thing, uh, we can say uh what are we doing? SUBSTITUTE. We can do SUBSTITUTE, we can do open parentheses, we'll give the range, and now let's say we want to change all of them to a different format. So instead of the um forward slash, I'm going to keep calling it that, if hope that's correct, we want to give it a dash, and so then we close that, and now all of them are in this new format. So it it's able to substitute a specific value for a new value, and if you don't include an instance then it'll do it to every single one in there. So let's go over here, and we're going to actually use the the um the uh the instance num, and I'll show you what that does, uh, and so really quick we'll do the exact same thing that we just did. We'll do the forward slash, and we want to replace it with this one again, this Dash, but we only want to do it on the first instance of that forward slash, and so as you can see all the ones that um all the ones that were replaced are the very first instance, whereas the second instance, which is the second time it appears in this string, does not get touched. So if we take this and we put it right over here and we move it to two, it's kind of the opposite. So the first one wasn't touched, the second one was. So we're choosing which instance or which time it shows up in that string, and then it replaces it. If you do not choose an instance, it chooses all of them. So this can be super useful if you want to do like a bulk replace, um, but you only want to do it on a specific column, um, and you just want to use a formula really quick, right? Um, and so you can use this in a lot of different ways. So that's how you're able to actually do it with the first instance, the second instance, and if you don't include an instance at all.

Let's go over to the SUM. Uh, this is one I think everyone knows how to use, but I want to show you two other ones um as well. So let's go to the SUM, and we're just going to do equals the SUM, and I hope you know what this is, well not hope, I if you don't know what this is, it just adds up all the numbers in a Range. So we're going to add, SUM means add, so we're going to take this, and it's going to give us the uh what all these salaries are together. So super super simple. SUM is one of probably the most basic formulas that you can do.

SUMIF is a little bit different. You can add an IF statement, which we learned right back here. You can add an IF statement and then add it if it meets a certain criter criteria. All right, so we're going to do equals SUMIF, and then you're going to need to give a range in criteria, and you can include a sum range if you would like. So we're going to do the salary again, we're going to do a comma, and now here's our criteria. Let's do if they have greater than 50,000 for their salary, and close our parentheses. So now it's only going to add up if their salary is greater than $50,000. Now his is 50,000 exactly, so that won't count, but we have 63 and 65,000, which does equal 128,000. So it it just gives a specific criteria or an IF statement, then it does the addition. Uh, so super useful on that one. So that is how you do a SUMIF.

And SUMIFS is kind of the same thing as we did back here. There's the IF and the IFS. So the IFS is going to be if it has it meets multiple conditions. So let's take a look at that one. So so let's do um equals SUMIFS. Now uh oops. Now the Syntax for this one is going to be a little bit different. You'll see that in just a second, but this adds the cells specified by a given set of conditions or criteria. So let's do an open open parentheses. We'll give the sum range. So let's do um the same one as before. Then we have our criteria range. So what are we looking at? What's um this is the area that's going to be added after all these IF statements are done, right? So we have to initially set that. Now we're going to say okay, what criteria are we basing this off of? So let's put a comma, and we're going to base it off of let's do this one. We'll say um if the uh gender, so we'll do comma, if that's female, oops, if that's female, and then we'll give another one. We can say if they're female and let's say they are greater than oops, greater than 30, and we'll close that up, and it's going to give us 88,000. So female female uh there's one two right here, so it's going to be this one and this one, and that equals 88,000. So that's how that works. You're able to incorporate several different conditions into uh the SUM formula.

So again, I know this one's super simple, but you you can use it much more complex way if you use the SUMIF and the SUMIFS. Um, almost the exact same thing for this COUNT. I'm not going to go super in depth into this one. Um, I'll just kind of show you because COUNT is um COUNT and SUM are kind of on the same level of difficulty. They're both pretty beginner. This is just going to give you a count of how many cells um are there. So let's give this range, um, and so it's not going to add it, it's just going to give us a count. So if we do right here and scroll over them like highlight them, this countdown here, oops, this countdown here is nine, and so it's going to give us that count. But we can do a COUNT with conditions exactly how we did it in the SUM. So if we do COUNTIF, Oops, I did not spell that right, if we do COUNTIF, we're going to give a range in a criteria, exact same as we did before. U so let's do this. I mean you can do this on basically any of these; it doesn't really for this demonstration it doesn't really matter, um, but we'll say if their salary is greater than 45,000. So how many people this is going to give us how many people have a salary over 45,000, and that's five. So before in the SUMIF, if we did that um we did 50,000, it adds everything together; the COUNT is just going to count the amount of cells that meet that criteria. And again, COUNTIFS uh we're going to have a criteria range, and then we will specify what IF statements we want to be uh to occur in order to count those cells. So let's do we want, you know, we want to count, let it can be any range or it can be any of these. We'll do the ID this time, and now we can say, you know, we want it to be is our criteria one, we can say we want it to be greater than, we want their ID to be greater than 105, and let's say we want them to be male. So they have an ID over a certain um a certain range, and then they are a male. So there's only three people that meet that criteria, and so it'll be Michael, Stanley, and Kevin. Those are our three people, and so it gives us a count. Very useful to give quick numbers like this. Something I I genuinely use a lot, um, and I know I've said that a lot during this tutorial, but that's because everything I'm showing you are things that I've used a lot, so I don't feel like um, you know, I'm speaking out of turn here.

Let's look at this one. This one is very um has some specific use cases. Um, notice that this is a text right now. Um, if you do it when it is uh in a date format, it actually will not work. I mean I can you can test it out yourself; you just got to trust me, it's not going to work. So so what this does is it's going to give you the range from this day to this day. That's what it's going to do. So let's do uh oops DAYS. It's going to we want to choose our end date. So this is our end date; that's kind of backward from what you think, end date to start date, you think start date to end date, so you have to start with this one, and then we're going to choose the start date, and now it's going to tell us how many um how many uh days was it from here to here, and this one it's 5,56.

NETWORKDAYS is extremely similar, except it takes out holidays and it takes out weekends, and you can see how many working days has this person um how many working days or NETWORKDAYS has this person worked, not including, you know, weekends and holidays, have they actually worked since their start date and their end date. So let's do NETWORKDAYS, and we need our start date, our end date, and you can specify extra holidays if you'd like, but there are a already standard set holidays in there that it takes out, um, so you know if you want to do that you can. So we're going to do the start date again. This one's different; this one says start date, end date, and then we're going to give the end date, and if you notice they are going to be different numbers, dramatically lower because it's taking out weekends and holidays. So this is how many days uh calendar days they've worked, and this is how many days they've been in the office and worked. And that is it. Um, again, there are so many formulas, I mean literally hundreds of formulas that you can utilize and use and are out there for you to try out yourself. If there are specific ones that I did not cover in this video, please put it in the comments below so that I can, you know, show you how to do these things. I I I will say I've probably used a majority of the ones that you're going to put in the comments already, and if I haven't used it, I'll take a look at it and see if it's really useful, and I'll show you that. So thank you guys so much for watching. I hope that this has been helpful. I I feel like a lot of these things are not things that I learned before I started. Almost all these are ones that I learned while I was on the job, and so I'm hoping that you can get ahead of the curve, and you can learn these things before you actually start, so that when you get in there you're just like killing it with the formulas, and people are like whoa, this guy is like this guy knows what he's doing in Excel, give him all the Excel work, and then you become like, you know, just the Excel guy, um, and everyone, you know, loves you for it. So with that being said, thank you so much for watching. I really do hope this helped. If you like this video, be sure to like and subscribe below. I'll see you in the next [Music] video.

What's going on everybody? Welcome back to another video. In this Excel tutorial, we'll be looking at XLOOKUP. Now if you don't already know what XLOOKUP is, it is a new feature in Excel to kind of replace VLOOKUP or to be a much better option, at least in my mind, as a much better option than VLOOKUP. And so if you're someone who's either used VLOOKUP a lot and you're trying to, you know, learn this new Option, or if you've never used it before, this video will be super helpful because I'll walk you through kind of the options and what XLOOKUP can do, as well as the different between XLOOKUP and VLOOKUP. But before we get into the tutorial, I want to give a huge shout out to today's sponsor, and that is Udemy. Udemy is the go-to place if you want a full-fledged course in Excel. I have three options of courses that I have taken on Udemy, so I'd highly recommend checking those out. They are having a huge sale on all their courses during this time, and so if you are in the market for a course, I highly recommend checking out Udemy and getting one there. Now without further ado, let's jump on my screen and start the tutorial.

All right, so let's get me off the screen because we all know why we're here. So I didn't include this in the the formulas video last week because I knew this was going to be a large one, and a lot of people are going to want to know how to do this, what the difference stream VLOOKUP and XLOOKUP is. So it has its own dedicated video to it. So let's get started. It is a Formula, so we're going to come in here in the cell, we're going to hit equal, and then we're going to start typing XLOOKUP. Now I'm going to hit tab in just a second, but let's read what this says. It says searches a range or an array for a match and Returns the corresponding item from a second range or array by default, and an exact match is used. So really useful to know. Um, we'll talk a little bit more about that in just a second. Let's hit Tab, and it's going to complete it, and it's going to start giving us or it's going to tell us what our input values need to be. We're going to have our lookup value, we're going to have our lookup array, our return array, and then some optional things like if not found. So if your option isn't found, you know what will be um you know the the uh output that it gives us, a match mode and a search mode, and I'm going to show you um kind of how to use every single one of these things. As you can see at the very bottom, I've kind of already set up all of the instructional um instructional content for this video, and so we'll kind of get through all these different scenarios. So let's just start really quickly with um how to use it very simply with the lookup, lookup array, and return array. So we're going to come in here, and we're going to give it our lookup value. Now Toby Flenderson right over here, and A3 is going to be our lookup value, so that's who we're going to be searching for. Now we're going to hit comma, and now we're going to be needing to look up uh or to input our lookup array. Now an array is just uh, you know, a range basically. So we're going to do this is where it's going to be searching for um that value. This is where it searches for A3. So here's Toby Flenderson, here's Toby Flenderson. So it will find it in this array right here. Then we're going to hit comma, and now we need to give it the return array, what it's going to return on that row when it finds it. So we're going to return his email, keep it really simple. So what it should do, and let's close this parentheses, what it should do is it should take Toby Flenderson, it's going to search in this column or in this array, and then it's going to return the email when it finds Toby Flenderson. So it's on Toby Flenderson is on row six, so it's going to find Toby Flenderson, it's going to come over here, and it's going to return Toby Flenderson dundermifflincorporate.com. That's what it should do. Let's see what it actually does. Let's hit enter, and it returns it. Now if we drag it down like this, it'll apply it to all of these names right here, and it works exactly how it's supposed to. Um, again, if you have never used VLOOKUP, you don't know how good you have it. Okay, VLOOKUP um was extremely useful, but just uh a bit complicated, and I'll talk about that near the end of the video when we compare VLOOKUP to XLOOKUP, but just know that if you're using XLOOKUP for

The first time and you're just getting into using Excel, you guys have it good. Okay, so just know that. Um, now let's go over here to X lookup multiple rows because you can return more than one output with, um, with X lookup. So let's go right in here, and we're going to basically write the exact same thing as we did before. So let's write X lookup. We're going to do Toby Fenderson as our value. We're going to search here, and we're going to do something a little bit different this time. We want to include our end date and the email. So what we're going to do is we're going to start here; we're going to go down all the way to the bottom of end date, and then we're also going to include the email. And when we do that, it will, uh, in in the output, give us a row or a column for end date and a column for email, so an output for both. So let's hit enter, and now we can see that we have the end date here and the email here.

Now, one of the downsides, or or something that I'm not a huge, huge fan of, is, well, first off, I love that you can do this; that's fantastic. Um, but it has to be right next to each other. So you you're only going to get that output exactly how it is in the columns. So if I went and did this range, um, I would include all of that. Um, so you know, let's just, for example, let's pull that down here. So let's take this and put it right here. If I did, instead of O2 to P10, if I included H to email, this whole range, and I hit enter, it's all going to be included. So you know that's one of the small downsides of of that functionality of when you can use multiple rows is that it's going to use the rows exactly as they are. You can't really customize it within the formula. You can move around, um, these columns to how you want it. Um, so that is something to note. And again, you can pull this down, and it'll be applied to all of those names.

Let's go over to X lookup exact match. So let's open this up. We're going to do equals X lookup as we've been doing, and we're actually going to be looking at the if not found and the match mode, um, both, you know, on this tab right here. So let's do what we've been doing before. We take our value that we're looking up; we take the, um, array that we're looking, and we're going to do the email. And, you know, as you can see, this says Toby Flender, not Toby Fenderson. So what we are going to do is we're going to hit comma, and if it's not found, you can return, um, a value or a string that you want to return. Now, for simple purposes or for simple instructional purposes, we're going to do not found, and then we're going to close that off. So let's do this, and Toby Fenderson was not found, and so it was returned not found. If Toby Fender was actually in this full name, then it would have returned the email. And then if, along the way, you know, one of these was not part of it, then, you know, we would have, uh, we would have had the not found.

All right, so let's go right up here. We're actually just going to copy this, uh, because I want to reuse it, um, and then we're going to go right here, and we hit a comma. Now, this is our match mode option, and so we have four different options that we can choose from. A zero is an exact match, and that is by default; that is what we have or what we use. Then there's a minus one, that's an exact match or next smaller item. Then there's a one, which is an exact match or next larger item. And then there's a two, which is a wild card character match. Now we're going to do that, and we are going to, um, you know, try this out, and it's not going to work, and not just because I forgot to put A4; it's doing it because it's searching for Beasley, but if there's not a wild card option already put in here, um, it doesn't recognize it. So we need to indicate where that wild card needs to be. So we're going to do a double apostrophe or quotation marks; we're going to put an asterisk right here and then do another one, and we're going to hit an ampersand. So we're going to have an ampersand right here, and what that's going to say is anything that comes before A4, anything that comes before Beasley is okay; doesn't matter what it is, as long as it has Beasley at the end, that is going to be okay. So we're going to have Pam that comes before Beasley, and that's going to tell it, and it's going to say, okay, I know that anything that comes before Beasley is all right. And so when we hit enter, is now going to return the output that we are looking for, and we can include that on these as well. Now, this one is Meredith, um, and so Meredith is at the beginning, so we have Meredith Palmer. So we can actually take this, and we're going to put this at the end, put the ampersand right here, and now it'll work. And the exact same thing for Kevin Malone right here, Kevin Malone. So it just didn't include, uh, the ne at the end, and so it's still going to work if we include that asterisk at the end.

Now, I know I said we were looking at search order, but I'm actually going to kind of give you an exact match, uh, first, and then search order, but it's just kind of easier to show it over here. So I'm going to do X lookup; I'm going to look up this value, do a comma; here's the range; this is our start date that's it's going to be looking for, and I want to return the full name. Now, no value in here has 112000, but what we can do is we can do comma, and then a comma for the match mode, and do an exact match or next larger. And I know this is in the exact match part, but it, you know, kind of refers to search order a little bit, um, where it searches for the next largest value; that's what that's what that number one represents, the next larger value. So we have 112000, and if we look right here, the next value above 112000 is 152000, and so it should return Angela Martin. Let's see if that works, and there it is.

Now let's look up the actual search order. Um, so let's do equals X lookup; this is the value that we want to be searching for, and we're going to be looking in this start date, and comma, and we want to return the name. Now let's get over to search mode. Now, the search mode performs a search starting at the first item, so at the very top, going down. So by default, it searches from first to last, but you can reverse that and do search from last to first, or we can do a binary search, which is where it sorts in ascending order or sorts in descending order, um, and that's with the actual value. And so we won't be able to show this binary search or on ascending or descending because our values are the same, but if we had different values and we were looking up, um, using this, um, next largest, we would be able to show that. But I'm going to show you the search from first to last and last to first. So let's put in, by default, and this is what it would be: search from first to last, what the default would be. So it starts at the very top; it goes down and finds the first 56 2001 and returns Toby Fenderson. Now, if we go in here and we hit minus one, that is going to search from last to first. So it's going to start at the bottom and go to the top, and the first one that it finds is Michael Scott. So that's that first one starting from the bottom, and then the Michael Scott right there. So these two, the exact match and the search order, can kind of be combined into, um, this one right here, we using this one, um, which is, you know, exact match or next larger, and you can include that in this binary search in this one as well.

All right. Now let's head over to the X lookup horizontal. I think we're, we only have a few left. Yep, X lookup horizontal, then we'll do X lookup with sum, and then I'm going to show you the V lookup at the end. So let's go right here. Let's say equals X lookup; the value that we want to be searching for is February; that's what we're looking for. Hit comma, and where do we want to search to find February? We want to search in, uh, these calendar months, and then we hit another comma, and now we're going to be searching for paper. So let's do paper, and we'll hit enter, and it found February, and it returned paper right here. And we can do that for paper, printer, and manila folders, and so it's going to give us the 310, the 40, and the 118 from February.

Now let's go right over here to X lookup with sum. Um, I actually, it's basically a carbon copy of this. Uh, let's take this over here real quick and place it right there because it's the exact same thing except at the end; we're going to use, I'm going to show you how to use sum with the X lookup at the same time. Now, um, we're going to be using the formula sum, and so we're going to do sum, and then within the sum, our first number is going to be an X lookup, and then our next value is also going to be an X lookup. So let's do X lookup, and now we're going to search for our very first value, oops, our very first lookup value. So we're going to go to I1, and then we're going to search this again, and we want whatever value, oops, goes into that. So let's close that parentheses, and now we're going to do a colon and another X lookup, and now let's do March. So now we're going to search for March; we're going to do our search range where we're searching for that March, and we want the paper as well, and let's close that, and then we also need to close that parentheses. So now we are basically adding this February and this March, so it's going to be 310 plus 150; it's adding those, um, two values, and it should be, uh, what, 460? So let's see if that is our output, and it is. So you can do this with a lot of things, not just sum, but you're able to use X lookup within different formulas. If you're searching for a specific value and a specific value, um, in in another, um, cell, you can add those together using X lookup, which is, honestly, it's pretty great.

So let's go over to V lookup. So I wanted to show you this because I wanted to show you where it came from and what we used to do, um, unless you are continuing to use V lookup, and what we can do now. So X lookup, I just showed you kind of everything, um, but super quickly, I'm going to show you how V lookup used to work, um, in a super short way so that you can understand how it used to be used and how it is used, uh, how X lookup is used now. So let's go in here, and we're going to say equals, and we're going to do a V lookup. And so we have a lookup value, and so we're going to click this; we're going to hit comma, just like we did before, and now we're going to do a table array, and the table array is is a little different in that you're searching an entire area. So let's do, uh, H2 all the way through O, oops, O10. So that's what that's what our table array is going to be. Then we're going to do a comma, and now we have to do a column index number, which number, um, are we going to be, um, searching for, which, um, value are we going to be searching for in here? And so we want to search for eight because this is 1, 2, 3, 4, 5, 6, 7, 8. We want to return that email, and we're searching for the name right here in this very first column. So we have that comma, and we're going to do eight, and then in the range lookup, you can do true, which is an approximate match, or false, which is an exact match, and we'll do false. I don't know why it's not auto, auto doing it, but there we go, and now we will do it, and it's going to return it just as we had it.

A lot of people, uh, I guess not everybody, but some people didn't like, and the reason why they created X lookup, you had to do those ranges, and if you ever went in here, and then we, let's say we, um, added another column, which happens to data, now it gives us completely different, different data. So let's say, for whatever reason, we added, uh, address. So now we have these people's address. Well, now it's going to give us a different, um, value; it's going to have this end date because if we go in here now, it doesn't, now the eighth is this end date, and the ninth is this email. So if you have a V lookup that you use for, um, you know, a calculation or a table that you've created or different things in Excel, you then have to go through here and manually change this. And so a lot of people didn't like that because if you, you know, needed to change data or you needed to change something or add an additional column, you'd have to go back and fix all of your V lookups; they wouldn't just automatically, uh, move with it, which is what happens with X lookup. And just to prove this, uh, let's go back to the very first one, which is the X lookup, and right now the email is looking at O2 and through O10. Um, we're just going to insert right here, and that would be our new column. We'll do address, oops, address, and notice that it hasn't changed. And why is that? Because it auto-changed for us from P2 to P10, understanding that it wanted to stick with when something was inserted here; it wanted to stick with the original data or the original array that was selected. And so X lookup does that work for you, and it makes it a little bit easier to automate things and create these processes in Excel without having to go fix it later, which you had to do with a V lookup. So that is it for today. I hope that you know how to use X lookup a little bit better now that you have watched this. Uh, if you enjoyed this video, be sure to like and subscribe below, and I will see you in the next [Music] video.

What's going on, everybody? Welcome back to another Excel tutorial. Today we'll be looking at conditional [Music] formatting. Now, if you've never heard of conditional formatting before, that's okay. I had never heard of it before I became a data analyst, and so now that I've been using Excel a lot, of course, I use it quite a bit, and so I want to show you how to use it. Conditional formatting is basically just a way to see patterns and trends in data, and that's a super simple way of putting it, um, but it's very easy to use, and so hopefully I can show you how to use it, uh, really easily, in a lot of the things that I use the most and some of the things that I use it for so that you can also know how to use conditional formatting. Now, before we jump into the tutorial, I want to give a huge shout out to the sponsor of this Excel series, and that is Udemy. You guys know by now that I absolutely love Udemy. I've been using them for years, and I've taken literally hundreds of courses on Udemy, and I've learned so, so much, especially when I was first starting out as a data analyst. Uh, I learned a lot through their Excel courses on Udemy, and so I have actually put the ones that I really like and I have taken and enjoyed and think you would as well in the description. So if you want to take those, be sure to check those out. Again, huge shout out to Udemy for sponsoring the series. Now, without further ado, let's jump onto my screen and get started with the tutorial.

All right, so let's jump right into it. On this Home tab right here, if we go all the way over to the right, there is conditional formatting, and the description that it gives us is easily spot trends and patterns in your data using bars, colors, and icons to visually highlight important values, and that is exactly how I would have defined it. A really good job, Microsoft; exactly how I would have done it. So what you'll see right away is there's nothing too complex. So we have some highlight cells rules, um, we have some top/bottom rules, data bars, color scales, icon sets, and then at the bottom, we can create a rule; we can clear the rule, and we can manage our rule. So if you create a rule, then you can manage it. So we're going to start with these icon sets, and I'm going to show you how to use those, and we'll work our way to the top, and then I'll show you how to create some rules yourself and how that all works. So let's start off with the icon sets. I'm going to go over here to sales, um, and for this data, we kind of have this, um, you know, trend or or pattern that you can kind of see over time, so over the months. Um, so if we go right here and let's use that conditional formatting, let's use that icon sets, and right here we can use these directional. So, you know, we have this kind of time series; each month that shows us how much paper they're selling, and if we do this right here, it's going to show us if it's kind of average or if it's below average or if it's above average or if it's going up. So at a really quick glance, you can kind of see the pattern of this data set; it's kind, kind of going mostly yellow and red; there's only two months where it's going up significantly. Now we don't have to only do that for one row or one column; you can apply to all of them. But as you can see, all of these are red. Now, why are they all red? It's because they're using numbers for everything, so they're comparing these 24s and these 50s and 65s against these 450s and 750s, and so they're all going to be red. But if we do it individually, if we do it each, each row, if we take it just like this and then we go to Icon sets and do it, it's going to be much more representative of the actual printers, not of all the numbers as a whole. And you can do other things; the arrows are ones that you'll probably see the most often; that's the one I've used if I ever do use them, um, but you can, you know, do ones like this where they have, you know, kind of a trend upward or a trend downward, um, and so there's just several more arrows. This one only gives you three, as you can see; this one gives you five, um, and you can do, you know, colors or shapes or or different indicators and all these different things, um, and honestly, it's kind of whatever you want to use; whatever makes sense for your data. But, you know, I've really only ever seen like these colors being used; I've never really seen these flags or anything like that. But again, it just depends on what industry you work in; you might, you might see that.

Let's go right over here to the demographics, um, and let's look at our color scales. Now, color scales are going to be the probably the most obvious thing that, in data bars, are going to be the most obvious things in here, um, if you go right here and and you look at this color scale, if it's high, if it's among the top ones, it's green; the lowest, it's red, and you can change that, um, to really any colors you want, any colors that they offer you, um, and it it does exactly what it does; it's a color scale, a gradient of the colors from high to low or low to high. And so any color that you do, you'll be able to kind of see, um, you know, what's good and what's not good. That really is, um, color scales in a nutshell. Data bars are again super, super straightforward; it's going to be either a gradient fill or a solid fill. So let's look at

The gradient fill, if we do a blue gradient fill. Actually, let's get rid of our um, let's go over here. Let's go to Clear Rules from Selected Cells. We haven't looked at that yet, but that's how you clear it. Let's go to Data Bars, and we'll use this blue gradient. So with this blue gradient, you know this one is, or sorry, this one is the highest one, so it's going to be completely filled. And this one is 36,000, almost half of this. I'm pretty close, and so it's almost half. Um, this one again, you know, it's not used very often. I, you don't see these a lot to be honest, you just don't. Um, but if you do see it, that's how you use it; that's how it can be done. Again, pretty easy, uh, as I just showed a second ago. If you want to clear the rules, you can clear from the selected cells; that's what we're doing. So I have column G selected, and I'm going to, I'm going to clear that. If you want to clear the rules for the entire sheet, you can do that as well, so it would affect every single column and row. We'll just do this for now.

So now let's go look at the Top/Bottom Rules. So this is the top 10 items, top 10%, bottom 10 items, bottom 10%, above average, and below average, and they're going to do exactly what you think they are going to do. If you select above average, it is going to select or highlight the cells that are above the average in column G. So let's look at the salaries that are above average, all right. And so, uh, the ones that are at the very top are Michael Scott, Toby Flenderson, and Dwight Schrute. No shock there. Um, I believe the average is somewhere around like 48,500 or something, so I think this one just is just below it, and so all these other ones are below average. And that's just because, you know, Michael Scott and Dwight Schrute and Toby are kind of bringing up that average quite a bit, so everyone else is going to fall beneath that. And so at a super quick glance, you're able to just highlight the cells, and you're able to see who is above average, and you know you can do this in a lot of different ways in Excel, but this is just a really simple, fast way to do that. Um, let's get rid of that real quick, and let's go back up here, and now we can, oops, let's go to Top/Bottom Rules, and now we can see the below average, and it's going to highlight all the other ones. And so it works exactly how you think it is going to work. And this is the default way that it highlights these cells, so it highlights them this kind of, um, see-through red, and then it highlights the actual text, or or the, um, characters in there red as well.

Now, I'm not going to go through and show you every single one of these Top/Bottom Rules. I think they're pretty self-explanatory. I just kind of wanted to show you what happens when you do use one of them; it's going to highlight that cell. So let's go up here to the Highlight Cells Rules, and honestly, these are the ones that I use by far the most. Uh, all these other ones combined, I do not use more than this Highlight Cells Rules. Um, and the one in here that I use more than any other conditional formatting rule is this Duplicate Values. So I'll start with that really quick, and I'll kind of show you a few of these other ones, but this Duplicate Values, to me, is one of the most useful ones. Um, and so let's kind of show you how that works. If you go to the Start Date, you can see that we have a duplicate value right here. And if we go over here to Conditional Formatting, Highlight Cells Rules, and Duplicate Values, it is going to highlight, um, the duplicate, and that says Duplicate right here. Now we can go through here and click on Unique, um, and then it would highlight all the ones that are not duplicates. Um, so you can use it, you know, kind of in a similar inverse way; uh, it's just different, different, but I use the duplicate almost always.

Another thing that you can do is go over here, and you can change the color, um, or you can even do a custom, um, which I just never do that; it's not, um, something I spend a lot of time doing. I typically just stick with this one. So you can do that, and it's going to highlight, um, you know, something that has a duplicate value in there. Now, why do I use this so much? Well, I work with a lot of different types of data sets, but one thing that you'll find in almost all of them is they have some type of ID, and they're going to have some type of, um, personal information, whether that's a Social Security number or an address or, um, you know, or a cell phone number or something like that; there is going to be data that is going to identify that person. Now, I work a lot with pharmaceutical data, a lot with Pharmacy data, um, as well as Healthcare data. So names, Social Security numbers, addresses, phone numbers, all those things, all that customer or or client information. And oftentimes, when I get a new data set and I have it in Excel or I convert it to Excel, I will start using these duplicates to try to find issues with the data, and I find them all the time. Either there's an employee ID or some type of customer ID or client ID that has a duplicate in there that should not be in there, or there's multiple Social Security numbers, or there's an issue in some other way, and I'm able to find those things and spot those patterns using this duplicates. And I promise you, I use this one almost every single time I open a new data set or I work with new clients working with their data. Um, and so I wanted to show you this one; I wanted to really press upon you that this one is a really, really, really good one to know and learn how to use. It's not complicated; it's not hard; it just shows you, you know, you know if there's a duplicate value. But I wanted you to know how I use it and how often I use it so that you can, you know, pick that up and put that in your toolkit and your back pocket so that you can use that later on if you have, uh, if you have a similar need or if you're trying to do something similar to what I was just talking about. So that is how duplicates work. Again, super great; it's obviously not super useful when you're only using, um, 10 rows, but when you have, you know, 50,000, 100,000, and there should be zero duplicates in there, and you highlight it, and then, uh, you come right here, use the filter, and we're going to filter, and we're going to sort by the color, and it allows you to sort by the color, and you have duplicates in there, then that's a problem, and you identified a problem super quickly, uh, and you know some of those things, they slip by because nobody checks it. And so that's something that I, I often check. And if you go here and you sort by color and there isn't an option to do, um, this, this pink/red color, and that means there aren't any duplicates, and that's a really good thing; most of the time that's a really good thing. So let's go ahead, and we're going to clear that as well as get rid of our conditional formatting rules.

Now, another one that I use a lot is this one right here, which is the Text that Contains. Honestly, this one comes a lot in handy, especially when you're looking for like a specific keyword. In my, uh, case, a lot of times I was using this when I was going through drug names. I am not a doctor; I do not pretend to be a doctor. And so when I was looking for Lorazepam or something like that, um, I would just search for like Loras or something, and and not Lorax, but Loras, you know, I, I would just search for it, and then all the ones that contain that would pop up; I can bring them to the top, and I can see them. And to me, that's super, super useful, and I would do that all the time. And so in this case, we're looking at emails, and let's say we all only wanted to pull all the ones that are Gmail. And so now we can go through, and we can, you know, click OK, and that's going to pop up, or we want all the ones that have Dunder Mifflin. And if we click on that, all the ones that are Dunder Mifflin come up, or have Dunder Mifflin in it. And again, we can, um, sort by, or we can, um, and so we can sort by right here, and we can bring all those to the top. And so super, super useful. Um, and another use for it that you may not think of is something like if it's, you know, there's some incorrect data in there. This happens often with phone numbers, addresses, um, start dates or or or dates in general, date formats where you can go in here, and you can say Text that Contains, and if you know you put in a, oops, a dash, and it has it in there, then you know that that is, that is wrong.

Now, that is really all I wanted to show you in the Highlight Cells Rules. Uh, the Duplicate Values and the Text that Contains are by far the ones that I use the most. All the other ones I have used, um, these ones not so much, but in these Highlight Cells Rules, I use, you know, these two all the time. Um, sometimes I use this Between; I don't really use these other ones as much, although I have used them. And so if you got nothing else from this video, I just wanted you to know that these two are super useful, and if you haven't used them before, to maybe try them out and see how you can apply them to your own data sets.

Now we've looked at all of these preset ones in conditional formatting, but you can also do a new rule. And so if we click on New Rule right here, and we go down to Use a formula to determine which cells to format, we can add our own formula in here that will then highlight exactly what we want. And so if there isn't a preset rule that you like and it doesn't have the option that you want, you can do almost any formula that you want. In our formulas video that we did a few weeks ago, and you can put it in here, and then you can format, uh, what you want the cell to look like if it meets that criteria. So let's take this right over here, um, and before we start this formula, I just want you to note that, you know, uh, I have H11 highlighted; that's going to come into play in just a little bit, but I want you to be aware that H11 is the cell that we're highlighted. So what we're going to do is we are going to create our formula. Now, if you've never created a formula, I highly recommend, uh, watching my formulas tutorial because that is going to show you how to do this, um, but we're all we're going to do is we're going to do equals; that's how you start the, uh, how you actually create a formula, and we're going to give it this range right here. And so, so it's going to take everything from G2 to G10. Now these dollar signs are super important. If you don't know how to use them or you don't know what they do, um, you're going to mess up this formula a lot, uh, and so what this dollar sign basically does is it's basically hardcoding it in there; it is only going to look at G2 and is only going to look at G10 or through G10 because that colon, and this can come into play because if you have something selected like the H11, it's going to mess it up because now if you have H11 selected like we do, you'll see this in a second; it's not going to be applied to this, um, and again, I'll show you that in just a minute, but we don't want this hardcoded in there. Okay, but we do have to select the proper range in a second, um, so we're going to get rid of this; we're going to get rid of the dollar signs because we want it to be pretty fluid and be able to apply to be applied basically anywhere we want. Let's go into this formula, um, if it meets our criteria, let's give it, um, let's give it a border, and we'll give it, um, we'll give it some color. We're going to say if this is greater than 50,000. So let's hit OK, and nothing happened. So let's go back and see why. So if we go to our Manage Rules, you can see that, so as the G2 to G10 is greater than 50,000, but it only is being applied to this H11 cell, which really makes no sense. Um, so if we had wanted to, to get it done the first time, we needed to have basically selected that G2 to G10 right away, um, but we can do that now. So let's get rid of this, and we're going to say G2 to G10, and that is hardcoded in there; that's should be fine still, um, but let's see what it does. And so now every single thing is highlighted. And why is that? Uh, that's because when we changed it, it also changed the format of it because we changed the cell that we were looking at. So we need to come back here, and that's why again you want to do this the right way the first time. We're going to come back here, and we're going to give it this range, and we're going to get rid of these dollar [Music] signs, and now we're going to hit OK. And so now it's being applied G2 to G10, and G2 to G10, and we'll keep it like that, and we'll apply it, and now it works properly. So now everything that's above 50,000 is being highlighted. Again, if that was confusing, um, it it is confusing; it genuinely is. And so if you wanted to do this right the first time without having to make a bunch of changes, you'd want to highlight these before you start, and then you want to go in and create the rule. We'll do this really quick just to kind of show you what I'm talking about. We'll say equals, we'll give it this range, get rid of these real quick because again, I don't want this hardcoded in there; it will ruin our formula. And then we'll say greater than 30, um, and we'll give this nice green, uh, and so now if they're over the age of 30, it will be highlighted, and we didn't have to go back and change anything; we didn't have to go back and fix anything like we did in the first one. Um, that was all for demonstration purposes, but again, you need to really be aware of that; that is something that I think almost everybody's going to mess up at some point. If you don't already know about it, then you definitely are going to make that mistake. Now, if we come over here, here in this area, uh, we go to our Manage Rules, and not just the current selection, but this whole worksheet, then you can see that we have these two formulas. Now you can go in and edit any of these by double-clicking or clicking on it and then hitting Edit Rule. You can also delete these rules or duplicate these rules. Um, I just wanted to show you what you are able to do with them, but if we, uh, go ahead and we get rid of this, um, so let's say we delete that rule and we hit Apply, uh, you know, the rule is going to go away; that's that, I mean, it's as simple as that. So that is how you can create your own rule. I want to be again very specific in the fact that that is a confusing piece, and if you mess that up, you're going to be, you know, fixing a bunch of different stuff and not understanding why your rule is not working properly; it's just because it's confusing. Those dollar signs are, are really important to watch out for. And that is all there is to it with conditional formatting. Again, conditional formatting is, um, you know, it's not anything super confusing; we've looked at more complicated things, but it's a really, really useful tool to use to look at these patterns and trends super quickly and to find, um, these outliers or these specific values that you're looking for very quickly. And if you're looking at just thousands and tens of thousands or hundreds of thousands of rows, this is one of the fastest ways to find these things without having to kind of wait and filter and use these, um, these filters right here because again, this can just take forever. Um, and so if you haven't or if you've never worked with a ton of data and tried to use this before, it can take honestly like 10 minutes for something simple that you could do with conditional formatting in like 10 seconds. So definitely something to mess with and use when you are working with your own data sets. Uh, I hope this was helpful. I mean, honestly, I use this all the time, so you know, I hope that somebody out there can, can use this, uh, for their own work that they're currently using. Thank you guys so much for watching; I really appreciate it. Again, huge shout out to Udemy for sponsoring this Excel series. If you like this video, be sure to like and subscribe below. I'll see you in the next [Music] video.

What's going on, everybody? Welcome back to another Excel tutorial. Today we will be looking at [Music] charts. Now, if you have data in Excel and you want to visually show that with bars or graphs or anything like that, you can do that really simply, and I'm going to show you how to do that today. And a lot of people are a little bit intimidated because they think it's a little bit complicated, but I promise you, by the end of this video, you will know how to do it like a pro. It's not that difficult; it's just you need to know where to look, where to click, and how to actually filter through things to make sure that you're visually showing the things that you want to show. But before we actually jump into the tutorial, I want to give a huge shout out to the sponsor of this Excel series, and that is Udemy. You may not know this, but I probably get at least 15 to 50 companies every single month reaching out to me wanting to sponsor the channel and promote their product, and I turn down almost every single one because I either don't know their product or I don't believe in their product. And so I'm not going to, you know, go and promote that on my channel, but Udemy is one that I have consistently promoted over the past year, and that's because I truly believe in their product. I've been taking courses off their platform for years, and I've honestly learned so much, and I cannot recommend them enough. So if you want to take a full-fledged Excel course, I have my recommendations in the description if you want to check those out. Thank you again to Udemy for sponsoring this Excel Series. So without further ado, let's jump onto my screen and get started with the tutorial.

All right, so let's jump right into it. Right here we have the Dunder Mifflin sales report, and over here we have all the products that they were selling along with the months that they were sold in. And so in January, they sold 450 reams of paper. Down here we have the total items per month, and so in January, they sold 898 units of, uh, products or or things that they sold, and at the very end, we have the year-end total. So this is the total amount of paper that they sold throughout the year. Now we're going to use this data right here for all of our charts. Now you may not have data exactly like this; it can come in lots of different flavors, but you're going to get the basic gist of how to use charts, how to edit it, how to customize it to fit what you need, and then we're going to kind of put it right over here and kind of create its own sheet where we can kind of visualize all the things that we want to show. So let's jump right back over here into Sales, and first thing we need to do is kind of highlight the data that we're going to be working with. Now I'm going to start with everything, but, um, you know, I'll show you along the way; we don't actually want everything, but we can filter that stuff out as we go.

So let's go right here, and we're going to insert, and we're going to go over to charts now. This is the chart section; there's lots of different types of charts, um, but the first thing that we're going to be looking at is right here. This is a 2D column, or kind of like a bar chart, and we're just going to click right here and we're going to pull this down.

So now that we have this down here, there are a few things that I want to show you before we actually really get into it, and I kind of want to show you the options that you have. So if you go up here, we have different, uh, chart Styles, and so if I hover over them, you can see that each one kind of looks a little bit different, and it really doesn't matter; it doesn't really change the data in any way, just how you visualize it. And so if that is important, if that is something that you, um, you want to stick with a certain theme or a certain look, then go for that.

The other thing that's really nice to have over here is this switch row and column. So right down here, you can see this purple, and you can see this red; those are our rows and columns, and we can switch that right here. So if we go like this, now instead of the months being right here, the months are the colors, and the actual product is right here. Let's click it again, and it'll go back. And so now we have this kind of Time series; now we have January through the end of your total.

Now, this one is one that I think is super helpful; you know, it you can do it down here as well if you go to this filter, um, but both of these are super helpful because you sometimes just want to select all the data and then kind of get in there and mess with it. Something that we want to get rid of is this total items per month, so we want to remove that, and then we also want to remove this year-end total because both of those are are kind of the end result; they're not the actual data per month or or per product. So we're going to get rid of those, and we're going to apply that. And as you can see, just right off the bat, our data is changed dramatically, uh, and that's because we aren't including these these large large numbers that were kind of throwing off uh the visualization for us.

So this one right here, as is, is already pretty good. Um, what we can do right here is we we can change this, and we're just going to say products sold per month. Now what we can do if we want to move it to another um to another sheet is we can actually move the chart, and we can select where we want to move it; we can move it to chart sheet, and we can do that, or something that I do um almost 99% of the times, I just copy, and I come over here, and I'm going to paste it. And so now we have this um this chart right over here as well as back here. And so I typically tend to do that because now we can still go over here and change this one as much as we want. So if we want to go in here, we can alter this one, and it won't affect the other one, so we just have basically two copies.

So we're going to keep this one right here; this is going to be our first visualization, um, and as I said, it's it's fairly straightforward if you've ever done any types of charts or graphs before. Um, right here it's January, February, March, April, May, and if you hover over these, you can see that that's the the paper, and if we just glance, you know, the paper is their biggest product by far, and so that blue, um, which is their paper, is going to be the biggest every single month, so that makes perfect sense.

Now, what if we want to change up uh the the kind? So what if we want to change up the kind of visualization that it offers us? Well, we have a lot of different options. Let's go right over here to change chart type. Now this is going to offer you just about everything you could possibly imagine or want, and even things that you absolutely would never ever want ever, um, and so I'm going to show you some of the good ones, and I'm going to show you some just absolutely insane ones that uh Excel came up with, which cannot I could not imagine a scenario that these are ever used, um, but within these columns you can do; they're called cluster columns, uh, these stacked columns, so would look just like this; those are often used as well, um, and then we have ones that they're just not used often. Let's look let's take a look at this one right here. I mean, it's tough; it's tough to look at, um, but let's let's put it right here. This is basically the same thing that we just had except visualized in a different, um, we'll call it more unique way, uh, and let's for the sake of it, let's put it over here.

These two things show the same information; they show the same data, just one is shown well, and one is not shown well. Um, I'm not a fan of these 3D type of visualizations; I I just don't like them, but maybe you do, and and you want to use that; that's fantastic. Let's go back.

Something else that you'll probably use a lot are things like these um these line graphs. Okay, so these are line graphs, and they're different types, so they're these stacked, um, 100% stacked line lines with markers, different flavors for this this type of line graph, and so you can go in here and take a look again. Um, not my favorite, but they have it as an option if you ch so choose to do this, um, but I kind of I'm kind of a simple guy, um, but I'm going to go in here, and it's pretty clustered. Um, I want to kind of take the ones that have the highest sales or the highest total amount sold, so that would be paper, manila folders, and three-ring binders. So let's go in here; we want to keep paper; we want to keep uh manila folders, and we want to keep three-ring binders, and let's apply that. And so now it's a lot cleaner, and we're just going to copy this, and we're going to put it over here, and I'm just putting these all over here for you U because we'll look at this at the end and just kind of see different options and and ways to do things as we have gone through this tutorial.

So let's go back here. Now something else that we haven't looked at is the actual colors and color schemes that you can do. So let's go right here to these chart Styles, and we can go to color. Now color is um something that probably is quite overlooked um in actual charts and graphs; some terrible colors like this or or this, um, where they're really close together, especially when you have a lot of them, um, for example, le let's just pretend we put all of them back really quickly; it is near impossible to distinguish these colors; we wouldn't we wouldn't want that. Let's go back to this color; you know, when you have it like uh in some of these colors, at least it at least distinguishes them, so you can kind of see what you're working with u but when you have it in these monochromatic options, sometimes they're just impossible to distinguish. So be sure to choose the colors that you're using so that if somebody who's never seen this data before looks at it, they can easily distinguish uh the product and the month that you are looking at. But let's go just back up here; we'll choose this default option. Um, well, let's choose this one right here; this one's nice, although there's lots of yellows and oranges. Let's see this one; this one's not bad; greens, blues, uh, and like yellows, so that's nice.

Other things that we want to look at, and there are these chart elements right here; other things that we can add are things like data labels, um, and right here it's super messy, um, but if we went back and we got rid of some of these things like the printer, staples, highlighters, pens, and total, we apply that; it's a little bit easier to distinguish, um, and that's, you know, something that you may be interested in doing. You can also add this data table at the bottom, which is the actual columns and rows that you have for this visualization right here. Now let's expand this quite a bit; I'm going to make this extremely large. If you have something like this, it actually can be pretty nice, um, you know, maybe we get rid of these data labels, but it can be easy because you're putting it all in one place. You can also make this two separate visualizations, so you can have one visualization just like this, and right underneath it, you can have the actual rows and columns, but this option allows you to put it all in one. So let's put this back down because that is way too big, and uh wait, let's expand it a little bit.

Now if you notice right here, we have our Legend up top; it is possible to actually change that. You can go right here, and you can move this um kind of wherever you want, um, but it's not exactly easy to put based off how we have it right here. If we go into this chart elements, we go down to Legend, and we hit this little arrow right here, we can select it on the right, the top, the left, and the bottom, or we can just go to more options, which allows us to push it anywhere, but um let's say I want to do it just like this; I'm going to put on the right, and I actually want to bring it down right here, and you know that's just an option if you want to kind of customize it a little further; makes a little cleaner; uh, you can do that with almost any of these things. So if you click on this, oops, if you click on this, you can move this anywhere as well. So if you want to move this over here on top of it, you can make it look terrible, or you can move it uh right back over here. You know, this is something that you can move around; uh, you just kind of want to make sure you're doing it the right way. So let's get this back where it was; there we go.

Now, before we go any further, let's copy that and put it right over here with our other uh charts and graphs. And if you see over here on this side, we have this format chart area. Notice I haven't showed you this at all yet; that is because I genuinely just don't use this almost at all. There are some good stuff in here, um, and I'm sure that you know if you were someone who really wants to go in there and super customize it, you can do that, um, but I honestly I just never get in here, and I never, you know, change the glow or the Shadows, um, just not something I use, and some of these are only for these three 3D formatting, which I never use, and so I'm not going to show you and walk through these things again. I I really don't use it, and so if you want to go in there and mess with it, uh, you know, by all means go for it; it's just not something that I want to take the time to show you.

And with that being said, let's go back over to this chart sheet that we have, and it was super super easy to get these um charts and graphs and and and whatnot. There are lots of different options again; if we go back here and we go up here to chart design and go to the change chart type, and again, there are a ton of different options like a pie chart, um, like this; it's it's, you know, you can try to figure this out and use these, um, but you know, I wanted to show you the ones that you'll probably use the most, which are these columns and line charts, and they all kind of are similar in their own way. This bar chart is basically, you know, this column chart just on its side, and so they all have their different flavor; they all have their different way of visualizing the data, but in essence, they're using the data in a similar way to to visualize it and represent the data itself, especially things like these box and whisker plots or these waterfall charts, uh, you know, these are things that usually require specific data to kind of use uh and and so I'm just using data that you'll probably see the most of, um, like this this sales data.

So I hope that this given you a pretty good, um, you know, quick understanding of how to use these, how to customize them, how to copy and paste them over to a different sheet to create some type of little uh chart and visualization sheet that you can use to show your employer and visualize the data that you are working with. Thank you guys so much for watching; I really appreciate it. Again, huge shout out to you to me for sponsoring this Excel series. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

What's going on everybody? Welcome back to the Excel tutorial Series. Today we will be looking at how to clean data in [Music] Excel. Now, knowing how to clean data in Excel is actually extremely useful, and there are a ton of techniques to do this. I'm going to be showing you the ones that I probably use the most; I feel like are the most helpful to kind of do the bulk or the majority of the data cleaning that you're going to do in Excel. Like I said, there's so many different ways and very specific things that you can do, but I'm going to highlight some of the bigger ones that I find the most useful, and some of you may be thinking, well, I'll just do my data cleaning in SQL or python or when I get it ready to put it in Tableau, um, but honestly, a lot of the data cleaning, at least a lot of the big stuff, I tend to do in Excel IF the data set is small enough to fit in Excel. And so I think it's actually really really useful to know how to do this because you'll most likely be doing it more than you think.

Now, before we jump into the tutorial, I want to give a shout out to the sponsor of this video, and is a brand new sponsor; it is unlocked by Z by HP. Unlocked is a movie that's actually broken up into four parts, and each of them have a unique data science challenge associated with it. Now I'm going to read this next part because it's extremely interesting: each challenge represents a different topic, so there's data visualization, text analysis, audio signal processing, and computer vision, and you can submit your answers in your work on their website for a chance to win one of 10 Zbook Studio laptops or a free trip to the Kaggle World Championships. So I'll leave a link in the description where you can go watch the movie and then do the challenges and then submit your answers for a chance to win. You should also go check out their hackathon where you can do these projects with other people just like you who are trying to figure out these answers and submit them to win as well. So go check that out. Thank you again to the sponsor of this video, unlocked by Z by HP. Now, without further ado, let's jump onto my screen and get started with the tutorial.

All right, so let's jump right into it. I have this US president data set; I got the base data set from Kaggle, uh, but I added some of my own data, and then I messed some stuff up as well, just to kind of demonstrate some of these things that we're going to be looking at today. This is not a full project, so you know we're actually going to be using this to create create any visualizations or anything like that, so you know all this is just for demonstration purposes, but we will be doing a full project and about two or three videos uh in this Excel Series where we're going to be doing from start to finish with a real data set, so you know if that's something that you're you wanting then we will absolutely be doing that.

Now, something that you may be wondering is how do you actually identify what you need to clean in the data? What do you know to look for? Well, some of the obvious things are things like formatting and standardization, so things like, you know, this James Monroe is in all caps; that happens all the time with in real data, um, and and so you know you want to standardize that, or this all lowercase; you want to standardize that; you want that all to be the same. There's also things like, um, right here where we have this wig and this wig with a bunch of random stuff after it; this happens all the time where it's not completely standardized, um, and you may even notice, um, you know, there are some spelling errors in here, and I'll we'll kind of look through that in a little bit. And then, you know, there are things like additional spaces where there shouldn't be spaces; there are things like currencies that you need to be aware of if you were importing this into or going to be importing this into a SQL database, um, things like currencies can be just a problem or be really um unnecessary; it may actually cause more issues in the long run, so you may just want to, you know, take that to the base uh value, and then dates are always an issue; always always always, um, so always look at your dates; make sure they're they're formatted correctly; make sure they're all the same. These are the types of things that right when I glance at this data set, these are things that I'm looking for.

Um, one other thing that is actually the first thing that we're going to start out with is you want to make sure that your data is not duplicated because if your data has duplicate data in it and you don't want that, it's not supposed to be there; there are some specific use cases where duplicated data is okay, um, you know, you want to get rid of that, and it's very easy to do in Excel. Uh, the first thing we're going to do, we're going to go uh to this data tab; we're going to go right over here, and we're going to get see if there's any uh duplicates in our data. So we're just going to go up to remove duplicates; it's going to automatically choose all of your columns to to check against, so it's going to for from a all the way through I; it's going to see is the exact same data in all these rows, and if it is, it's going to get rid of it, um, and so we're going to click okay, and it did find one duplicate; I'll show you that one real quick, um, because you know it was right here, so Barack Obama was here twice, and then I'm going to hit control; I hit control Z to go back; I'm going to hit control y to go forward, and it removed that uh that row completely. Now, in this example, you may be able to spot that with your eye, but in a real data set where you have 10,000, 100,000 rows, there's absolutely no way you're going to see that or very very unlikely that you are going to see that there's duplicated data in there, so just running a a quick um dup or or removing of duplicates that is really important to make sure that you have gotten rid of those things. So that's one of the first things that I do.

We're going to go into a lot of these different uh columns, and I'm going to kind of show you different techniques or things that I do when I look at actual data. So I'm going to come right over here; I'm going to insert, and this is what I actually do; I I usually create a separate column, especially when I'm working with this because I don't want to change this one; um, I don't want to go in here and you know say um equals upper equals proper, etc. There's a lot of different ways that you can change um names, or not a lot, but the main ones that you can change names, and all of them are completely okay. So, for example, I'm going to hit equal upper oops upper, and I'm going to go like this and close my parentheses. So I selected the cell; I close my parentheses; I hit enter; it is, and I'm going to hit

Um, in the bottom right, I'm going to double-click this, and it's going to apply it to all of them. It is completely okay to have your data like this if you want it to be like that. Um, if you want it to be all lower, you can do that. If you want it to be in proper case, you can do that. Um, there are—oops—there are different um uses for all of them, and honestly, as long as it's all the same, typically it's okay. But if, um, you know, for example, if you're selling this to like a third-party company or something like that, they may have um what they want for their ingestion process when they take your file in. If you send, you know, a weekly file or a monthly file, they may want it exactly how they want it, and you can change that to what they want. Um, but as long as it's standardized for you, it's all the same for you; that is a good thing.

So now we have all of these um in the proper case. That's typically what I do, or I use upper; those are the ones I use the most. I don't usually use um lower. And if you go in here and you type in lower, you know, it changes it to all lower. I don't typically do that. Um, and I'm going to add—I'm oops—I'm going to say "president-fixed," and so now all of these names, um, all of these uh different uppercase and lowercase, these are all fixed, and it just makes it so much easier to read, and you don't have different um uppercase and lowercase issues; it's all the same. So I'm going to keep that right there.

Uh, if we move a little bit to the right, if you look at this "prior," now this "prior" is a mess. It has stuff all over, and to be honest, this is not really something that I would probably be using um like in a real data set. I would look at this column and I'd say this is pretty useless. Um, if I had a very specific use case for this data in this column, I might try to, you know, parse it out and do something, but I don't. Uh, this is a completely useless column to me, so I'm actually going to skip this one. I'm going to go to this "party" one. In this one, to me, is looks pretty important because this is something that I know I can Group by, um, and I can create visualizations with, and kind of break that out. And if you look right here, we're going to add um we're going to add a filter. So now let's open up "party" and take a look.

So if we look right here, we have Democratic, democratic-republican, Federalist, nonpartisan, Republican, Republicans, Whig, and Whig with a date and some information in the back of it, and then some blanks. Um, and it's really important when we're when we're looking at these um ones that we think we might Group by that we have these um properly grouped. So Republican and Republicans, to me, right off the bat, looks like a spelling error, and so I'm just going to deselect All. I'm going to go to Republican, Republicans, and it's literally Republican all the way down except for this last one, and to me, that's just something that I would update. So I would just go right here; I do that. If I didn't do that, and then I try to create, let's say, a pivot table on here, I'll have its own group of Republicans, and it wouldn't be added to Republican, and maybe that's on purpose, but let's just presume that we know this data extremely well, and that's not supposed to be like that. Right again, that just comes back to knowing your data really well, understanding what it um, you know, what it should look like, and we know that it should not be like that, so we're going to fix that.

Uh, the next thing that we're going to fix, um, and as you can see, it got rid of it. Next thing we're going to fix is this "Whig." Um, that's just like an error; that's that's some issue on the data side, and we're just going to fix that by updating it, and that's it. I would always be keeping um a copy of this with the raw data uh somewhere else because this is presumably like a working document; this is not a um, you know, you aren't saving over your original file; let's just say that. And then let's take a look at these LS real quick. Um, okay, so there're these rows right here that have nothing. I think we're okay, but if we see anything different—47, 48—okay, so yeah, it's just these ones right here that have no data in it. Anyways, it's just seeing it in the filter, so not an issue at all. So okay, we're looking good. We've gone all the way over; we fixed this "President"; we skipped this one; um, we cleaned up this "party," and I kept this one in here because I'm not exactly sure sure if that's a Democratic or Republican, so I'm going to keep it its own thing. Um, I'm not a huge uh history buff in that aspect.

The next one right here is um the next one right here is really easy. Uh, this is something that happens all the time, especially on—actually, most often it's happens on numerical data—so like uh, you know, there'll be a number of 1, 1, and then there'll be a space after it for absolutely no reason, uh, and it happens all the time. It does happen like this as well, where you'll see this, and all you got to do is do "trim," and select the uh the cell. We're going to close that parenthesis, and we're going to apply that all the way down. What is so fantastic about the "trim" is that it's really intuitive, and it knows basically everything it needs to do. For example, um, it gets rid of the um spaces before it, gets rid of extra spaces in the middle, and um it gets rid of extra spaces at the end, um which you wouldn't be able to see, but they are there, and they they absolutely can cause issues if you have spaces at the end that you cannot see. Um, let's take this one for example; like if I had spaces at the end, that can cause issues when you insert or or or put that into a database. Um, that happens a lot with numbers, um, you know, when you're putting that into SQL; that can cause issues, and so you really it is important to actually do that "trim," um, and you can do that on all of your columns or just ones that you know you're having issues with. But once you import that data into SQL, you will know if there's an issue or not when you actually try to start using it. So we're going to say "Vice," and we're going to say "fixed." Oops, there we go.

Uh, this next one is one that you'll run into a lot when you're working with numerical data. You will encounter so many different issues. Um, one that I run into a lot is—I've worked with a lot of cost data or pricing data, and when it's in an Excel, it sometimes comes in with um these currencies like a dollar sign, a pound sign, things like that, and when you put that into SQL, it just is a nuisance, right? You're not going to be able to run—it's going to go in as a text, or it's going to be like a string, right, because it has that special character, and you don't want that. You don't want to have to then go in and then change things around; you just want to be able to start um, you know, doing calculations on those numbers. So what you can do is—sometimes it'll come in as a text, sometimes it'll come in as um currency, which I think this one's a currency—we are just going to change that to be a number, and then we're going to get rid of these—oops—and get rid of those. That it doesn't look as pretty, but that is much more useful than actually having the currency on there um with the decimals. This actually is so much easier when you when you want to use it for almost anything because you're able to add and do things properly in other systems. In Excel, I think it does understand it, um, but you know that can cause issues, so there is how you do that.

The next thing that we're going to look at is these dates, and just notoriously, whenever I see a date field, I know there's going to be an issue with it. It's very rare that I get a date field that is perfect. Uh, it just—it is genuinely is um is novelty when that happens, and most of the time it has to do with um, let's say, a date comes into Excel, and it's in a text format, or date comes into Excel, and they're not the same. In this example, they are not the same, um, and we just want them to all be similar. They say "date" on—if you look right here, it says "date," it says "date"; it looks like it should be the same, um, but if we go like this, it all looks the same, right? There's no issues at all. If we were to um try to use that, it may or may not be an issue, but we don't want to leave that to chance later on. If you're using this with Python or something like that, it can cause issues. Maybe not in SQL because it may um see the underlying um what's in the underlying cell, not just what we see, but some systems won't, and so you want to make sure that they're all the same, and so you know what we were doing back here with um oops with "party," and we were looking at this uh this filter and identifying the issues. I usually do that on date fields as well, and and often times—I know just for just for demonstration purposes—oftentimes I will get something like that, and then I'll come up here and I'll notice that there's this one random number that happens all the time, all the time, um, and so you know you want to make sure that you um that you look at these things and just just do at least a quick glance, if not kind of doing a kind of a deep dive into it. But all we're going to do is we're going to do both of these, and we're going to do a short date, and let's take a look and see if it fixed it. And so now they are all the same format, and that is fantastic; that is exactly what we want.

Uh, we're going to go back through here; we're going to get rid of these. Um, again, this is a working—this is a working document. Oops, we need to—we're—I'm going to do um Control+Shift+Down. Oops, let me go back up; do Control+Shift+Down and copy, and what I'm going to do right now is I'm actually going to copy—let me do it right here; I'll show you—sometimes I do this; it just depends. I'm going to go right here; I'm going to hit right-click, and I'm going to paste as a value, which means it's not going to take the calculation or the formula that I just did; uh, it's going to actually paste it as that value. So we just replaced it. Um, right here you can see up here it says "=TRIM(G2)". This now—now that I copied and pasted it over as a value—it got rid of that um calculation, and now it is actually a string, so we don't need this anymore. And I'll do the same thing over here as well. I'm going to Control+Shift+Down, copy, and I just hit the right key uh or the left key, sorry. Now I'm going to right-click, and I'm going to do "Paste as Value," and again has this "proper," and now it doesn't have the "proper"; it's actually the value that was here, so that's really important to note. Uh, and we're going to get rid of that one, and so now what we have is is already looking much better.

Now one of the last things that we're going to look at is deleting columns that we are not going to use, and this is why it's so important to keep a backup or or the raw data not in this file, because if you start saving over this file and this is your raw file, uh, that can mess up a lot of things, and that happened to me before, and it's terrible, and then you have to request another file, or you have to go back and find it or something like that; it's terrible. Um, so so this is our working document, so we can mess with this and do whatever we want for our purposes. Now for us, um, I can already tell you that this "prior" is a bunch of nonsense, and we do not need it; we're not going to use it for anything, and it—and if we have—this is a small, very small data set; this only has like um, let's say, you know, one, two, three, four, five, six, seven, eight; we have like eight columns that we're, you know, kind of using that has data—eight or nine. Now that's a small data set. I've had ones with literally like hundreds, um, and and it has so many columns, uh so much data, and sometimes it's good to just trim it back to the things you know you're going to use. This to me is absolutely useless. Um, we're going to delete that, and then right over here, it's pretty redundant; um, it's just one number off, but if we scroll down just a little bit, um it goes—it's basically just counts; it's a you could even call it a unique um identifier if you want; sure, why not? But we don't need both, um, so we're going to get rid of this first one. And now we have more of the useful and relevant data rather than the stuff that we absolutely know that we are not going to use.

Um, these "date updated" and "date created," we may never use them, but we might, um, so it doesn't hurt to keep it on hand. Those other ones are ones that we are almost certain we will never use again. Keep a backup just in case you need it; you can always go back and get it. So you know, if you go back to what we started with and you look at what we have now, it is much cleaner; it's much more usable, and these are small, subtle changes, um, especially with this very small data set of only like 50 rows or or 46 rows. But you're going to be working with data sets that are thousands, tens of thousands, hundreds of thousands of rows, and you need to know how to kind of look at this data, standardize it, um format it properly for what you're going to be using it for. If you're keeping it in Excel, there are different things that you may do than if you're putting it into a database or going to be using it in, you know, um using Python to to access it. So you need to kind of know your use case, but these are some things that I do all the time to kind of clean up the data before I use it for something, whether I'm creating pivot tables or I'm inserting it into or or I'm putting it into SQL. These are things I do all the time, and so hopefully that helps give you kind of an idea of some of the things that you should be looking for when you're actually cleaning data. And it's really important to understand why you're actually making these changes and the reason you're making these changes because some of the things that I did today may not be things you want to do on a different data set that has different uses and different um purposes for. So you know, take everything that I've said and and apply it um with a little grain of salt to your data set because your specific needs may be different than what I wanted when I was cleaning my data set. So I hope this was helpful; I hope you this gave you a small glimpse so some of the things that I'm looking for when I clean a data set or I get a new data set in and I'm kind of, you know, analyzing it, figuring out what I need to fix in it. I hope this has been helpful.

Uh, with that being said, thank you so much for watching. I really appreciate it. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

What's going on, everybody? Welcome back to the Excel tutorial series. Today we're going to create an entire project in [Music] Excel. Now, if you've never done a complete project in Excel where you take the data, you clean it, and then you create an actual dashboard where people can click on things and filter things, this is going to be a really great learning opportunity, as well as potentially, you know, a simple project that you can use for your portfolio, or you can spice things up and go a little farther than what we're going to be doing in today's video. I will walk you through every single step of the way, and hopefully we learn something together. And without further ado, let's jump right into it. Let's jump onto my screen and get started with the project.

All right, so this is the data set that we're going to be working with. I will leave a link in the description to my GitHub where you can go and download it so you can be working with the exact same data set that I am using. Now, before we actually get into this data and start looking at it, I'm going to show you what the final dashboard is going to look like. Um, we're going to create a few different types of visualizations—nothing too crazy—um, and then we'll create some filters as well, so we can kind of, you know, create some interactive filters with our data. So let's go right on over to our data set. Now I'm going to hide this because we are not going to use that, but what I am going to do before we do anything is I'm going to create a dashboard, and I'm going to create a pivot table, oops, and I'm going to create a working sheet. So um all these things have have different uses, and I'll explain that as we go along. So this is our data set. Um, I'm going to copy this over to our working sheet. When I go into, you know, an Excel and I'm working on something, I don't like to, you know, use just the one that I was using in case I mess something up and it saves over or some issue. I like to create a working sheet and keep the raw data right over here; it just makes my life easier. I don't have to save it and then, you know, open up a different Excel to compare them. So we have our "bike buyers"; this is our working sheet; this is our raw data; this is the one we're actually be working on today. So let's um let's start looking at it really quick and just kind of glance and see what data we're working with, and then we'll start cleaning it up, making it more useful for what we are going to be using it for, and then we'll start building out the dashboard. So right here we have an ID that should be a unique ID to each person; uh, this is their marital status—so married or single; this is their gender—male, female; have their income, children, their education, their occupation, do they own a home, how many cars they own, how long their commute is, the region where they live, their age, and if they purchased a bike. And this column right here is extremely important; this is going to tell us whether they did or did not buy a bike. So we got their information; they're looking for a bike, but they either decided not to buy a bike or they did buy a bike, and we're going to be using that one a lot in in this video. And so um, you know, this is basically the data set that we're working with, um, some of the demographics and and information behind the person. So what we want to do when we are cleaning the data before we do anything, uh, I like to see if there are any duplicates in here. Um, what we're going to do is come right up here; we can go to uh where is it? Right here, we got "Remove Duplicates." So we're going to click on that; it selects every single one; we just want to see if there's any useless duplicated data that we do not need; uh, and the data is a header, so we're going to click "OK." All right, so we had a ton of duplicates in there, uh, for whatever reason, so yeah, we do have duplicates in there, so I'm

Glad we did that. Otherwise, we would have, uh, you know, not good data, and we don't want that. Let's start right over here. Um, the ID, of course, we're not going to change. The marital status and gender are M's, S's, F's, and M's. Um, this isn't inherently a bad thing to have it like this, but you know, we have to think about it from the perspective of someone who's going to be using this dashboard. Do they know what M and S is? Do they know what M, uh, and F is? And if they don't, it's better to just spell it out for the most part. Um, so let's just do that. So we're going to click on the column B; we're going to hit Control+H. That's going to bring up our find and replace. Now, there's an M in both of these columns, and there's different things; one is married, and one means male. So we're going to do is we're going to search by columns, um, and we'll have match case. I don't think that's going to change anything, but that just means an exact match, uh, and we're going to do M equals, and we're going to replace it with married, and we'll replace all. Awesome. And then we're going to do S is single. This one is super easy. We're going to do the exact same thing right here. So column C, going to hit Control+H, we'll do still has by column, so we'll do M is male; we'll replace all of those, and F is female, and replace all those. That's great. Uh, you know, the next column right here is income, and in a, in a previous video, I talked about how I don't typically like it in this format, and that's true. Um, if you're doing calculations on it or or any other thing, it can mess it up sometimes. Having the dollar sign or it being a currency, we're not really going to mess with it too much right now. Um, what we can do is just kind of, we make sure all of it's currency; um, we'll just go like that to make it a little simpler, but we're not going to change it to like a numeric. Um, we will use this in the visualization; we'll see how it looks, and if we need to, we'll come back and change it; if not, we'll keep it how it is. Um, so that's all we're going to do to that one. Uh, the children, those look good. We have education, partial College, partial High School; this looks fine to me. Um, if there's any spelling errors or anything like that, of course, we need to clean that up. It doesn't look like there is. Occupation, skilled, manual, manual. Okay, those should be separate. Are they a homeowner? Should just be yes or no. All right, we have cars: one, two, three, four. Good night, who owns four cars? Um, and then that we have the commute distance, uh, and you know, there's nothing terrible about this. It's giving you ranges, um, which can be a good thing. I say let's keep it for now, but I have a feeling when we get further and we start using it in the visualization, we may want to change this, so let's just hold off for now. Um, but if needed, we will come back to this, and we'll change this. Um, and then we have our region, and that looks totally fine, and we have our age. Now, when you're using ages, typically you have some type of like age bracket or or age range, and you do that because there are so many ages in here, right? It's 25 all the way down to 89, and if you're using that in some type of visualization, it could just get really messy, and so you'll create kind of, you know, just brackets around these so that you can kind of condense it and make it a little bit easier to understand. So let's do that and just create a new column, and then we can use that for our dashboard. So let's go right up here; we're just going to create a new column; uh, we'll call this age brackets, and what we can do is we can use an if statement to kind of say if it's older than or less than and and and kind of give them these ranges. Um, that's one way to do it, and that's the way we're going to do it right now. So let's go up here, and what we want to do is we want to say is G; we're going to say equals; we're going to do IF, and we're going to close that parenthesis. Now, what we're going to say is if this, we'll go right back up here, if this is less than, so we're going to do this 31, and we're going to say comma. So if they are less than 31, what do we want to call them? What do we want their, their, you know, name to be? We'll call them adolescent. Oops, that's not how you spell adolescent. Adolescent. Um, and then if they're not, what we're going to do is we're going to say it's invalid. Okay, and let's just see if this one works first. All right, it's not working at all. Um, okay, so basically what we did was, um, incorrect; we did it backward. Uh, we want to do, I said, L2 is greater than 31. No, we want to do like this. So let's do that now. All right, and it should pull up where if they're under the age of 31, so if they're 30 or below is basically what it's saying. So if they're 31, they'll be invalid, but if they're 30 or below, it's adolescent, so it is working properly. Um, and let's see what it, see what it says. Perfect. So this one is working, and and now what we want to do is we actually want to build on this and make it, uh, kind of like a nested IF statement, if you've ever heard of that or done that before. So this is our first IF statement, and this is going to be, this is invalid; this is our value if false statement. This whole statement is going to be become our value if false for a different IF statement. Um, so let, let me write it out, and hopefully that'll make sense, but we're going to say IF, do open parentheses, and we're going to do it like this, and let's just get rid of this for a second. All right, uh, what did I do? And let me do, oops, give me a second. Okay, we have our IF; let me just write that out again. We have our IF; there we go. So now what we're going to do is we're going to write basically the next part of it. So we're going to say IF that L2 is, and we're going to do this time; we're going to do greater than or equal to 31. So now it's going to include that 31. So right here, we did anything less than 31, so it's 30 and below. This one is going to be 31 and above. So we're going to say these people are middle age, and if not, then it's going to go to this IF statement, and then we need to close it, I believe. So now let's try this. All right, fantastic. Now, if, um, everybody should be in one of these areas, right? Everyone should either be an adolescent or middle age because basically all we're saying is is if they're older than 31 or 30 or below; that's all these two statements do. So we have, um, you know, our next group. Now we can add and go even further into this, and now we can use this entire thing as the, um, what was it called? The value if false section. So that's what we're going to do; we're going to do one more. So we're going to have three different categories. So we're going to say IF and do, uh, an open parenthesis, and we're going to say IF, oh, actually, let's do it, um, let's not do it to this one; let's do it to this top one; just easier. Uh, so we're going to say IF, open parentheses, we're going to say L2, and this time we're going to say anybody over the age of 50, uh, or we can do 55; let's do 55. So we do 55, and we're going to call them old, and we'll do a comma, and this is the value if false statement, and we need to close the parenthesis. So let's try this. Anybody over the age of 55 should have old. Um, you know, maybe we'll do 54. So anybody who is 55 is considered old. I think that's fair. I think that's fair, guys. Oops, I should have done, I should have done that to this one. Let me get out of this, and we'll do 54. My dad is 55; that's why I'm doing it like this. This is free dead because he should be in this old category, to be fair. So now we have adolescent, adolescent, middle-age, and old. These are three categories, so we can now have these buckets, these different groups of ages, and it's much more usable than these individual ages, um, and so we will be using this in our, in our dashboard for sure. Now our next one is the purchased bike, uh, and we're not going to do anything with that. So you know, that is, that is that one, and you know, there wasn't a ton to clean up here. We removed some duplicates. Um, I don't know why it says that. What did I do? Married, married. What does this mean even mean? Did I write that? Did I mess this up, guys? Oh, when I did the M and the S, uh, replacement in there, it replaced it with married and single. It's supposed to say marital status. Oops. Thanks for catching that, guys. Thanks for catching that. I hope that's how you spell marital. Uh, we'll see. So, uh, we are going to keep it just like this. Now what we are going to, now, now what we are going to do is build pivot tables with this data. So we had our raw data; we have our working sheet, and now we want to create pivot tables, and pivot tables is how you actually help build your dashboards or help build your visualizations. So we're going to go right here; we're going to hit, whoops, get rid of that; we're going to go right here; we're going to insert, and we're going to say pivot table, and it's going to ask us what range. So we're going to go back to the working sheet, and we'll just click here, here, and hit Control+A. This is going to select all of our data for us, so it's really easy, and we're going to hit okay. And so now we have all of our pivot, I don't need, I don't need to pull it out that far; that was way too far. And now we have all of our pivot table information over here, and so that should make it really easy to, you know, actually build out. So what we're going to do is start selecting what columns and what data we actually want to work with. So the first one that we're going to build out is a dashboard that is basically looking at the average income of somebody who either bought or did not buy a bike. So we need, in this one, we're going to need their income; that's definitely going to be a value right here, um, but we want to break it out by male and female. So let's look at their gender; we're going to pull that down into the rows. So, um, this is basically a sum, and no, let's look at, let's make this an average. So I just went to the, um, I clicked right here; I went to the value field settings, and we're just going to do an average. All right, and then we are going to make these, um, and as you can see, there's four decimal points. Um, we'll keep it as is right now, but we may need to go back and change something. Then we're going to look at if they purchased a bike or not, and we're going to put that right here, so we can see that, uh, right here for the people who did not buy a bike, the females, their, their average salary was 53,000; the average salary for the, average salary for males was 56,000. For yes, the ones who did buy a bike, the average salary was 55 for female and 60 for male. So the people who had a little bit more money are buying bikes, and you can also see that, uh, the men are making more money in this data set just overall in general. Um, so let's make the visualization really quick, but you know, I don't know, I'm not a huge fan of these decimal points, and maybe we can just change that in the visualization; we'll see. Um, oops, that's not what I meant to do. Um, let's do that. So what we are going to do is we're going to click into here; we're going to click insert, and we're going to go to these recommended charts, and it's going to bring up basically every single type that we would want, um, and we can just click in here and see which one looks good. Oh yeah, I love those 3D ones; those are my favorite; you guys know that. Uh, let's, let's use this one right here; pretty simple. Um, whoops, let's pull this right over here, and as is, it looks pretty good. Um, you know, it shows male, female; we have the average or the incomes right here, whether they did or did not purchase it, um, and so at a glance, it's pretty easy to see. Let's see if there's anything, um, you know, if you want to change up style-wise, go for it. I'm just going to keep it as is, um, but let's see if there's anything we need to add. Right, right. Do we want to add these access titles? Uh, for the most part, I, I tend to do that; it makes it pretty easy to see. So we can go in here, and we can just click it like this, and we'll say income, and we'll say, we'll do gender, so that's what that is. And let's go back in here. Do we want to add a chart title? We definitely want to add a chart title, uh, for most of these. We'll add a chart title for sure. So we'll say average income per purchase. I don't know if that's 100% right, but we'll, we'll, we'll use it. Uh, if we need to change it to be, you know, by gender or something, we can, but, um, for now, let's see. Do we want to add data labels? Uh, definitely not. A data table, um, we can do this; it may make it a little easier to read. I will say that again; these numbers are just, these decimal points are really throwing me off. Let's go see if, um, we can change it in here. Let's go to see if we can just make these numbers. Okay, and, um, we can keep it like that, or we can even do something like this: add commas. Yeah, I'm going to keep it just like this. I, I think this just looks the best. Um, again, I'm, I'm getting, adding commas here; I'm changing the, um, decimal place right here; it just makes it look a little nicer, a little cleaner. Um, so let's keep this exact how it is. Um, we can always change things if we want to, if we want to come back to it. So we created our pivot table, and then we created our visualization, basically exactly what we're going to do for all of these because, again, all of these need, um, you know, all of these need pivot tables in order to create the visualization. So let's, um, get out of here; we're going to scroll down, and we're going to create our next pivot table, and once we get done with all of the pivot tables that we need or all the visualizations that we need, then we will, um, we will start. So we're going to do Control+A; we're going to do okay, and basically do the exact same thing that we did. Um, this time we're going to look at the distance. So for this one, I wanted to see, you know, I try to, you know, I created this already; I've already done this entire project through, but I haven't really talked about why or what we're going to look at for this one. You know, we're looking at is their income; does it change whether they bought or didn't buy one? Um, so, so if they said yes, you know, is there a reason? Are they making more money? Is, you know, are price points, are the customers, do they make more money, so you cater to them or not? Uh, that's a good question. Uh, another thing is, you know, we sell bikes, or this person sells bikes, so commuting distance definitely makes a difference. You know, does the person who is buying a bike live one mile away from where they work or 20 miles away? Uh, this will help us determine; this next visualization will help us determine, you know, who, who is doing that or who's buying it. So what we're going to do is we are going to look at the, um, that one that we were looking at earlier, the commute distance. So we're going to bring that right over here. So we have these, you know, one mile, 10 mile, 1.2, etc. Now we are going to, uh, again, we're going to look at if they purchased a bike; that's really important, and let's make that the column as well. So now what we have is a count of these Nos and yes, whether they did or did not buy a bike. Um, one of the issues I already see, and we'll, I'm going to visualize it, and then I'll show you that this 10 miles, you know, it's right next to the 0.1, so it's not in order, um, and that could be, that could be an issue. Um, so we may have to revise that somehow to put it at the very bottom because we can either do ascending or descending, uh, either one; I don't think is going to work, so we may have to work through that in just a second. Um, I don't, don't know if I did that in my plan for that. Um, yeah, so it has this big dip. Um, yeah, so let's, let's create it. That's okay; we're going to figure this one out together because I honestly, um, I didn't plan for this one. So okay, we have 0.1 miles; that's exactly where it needs to be; the one, the two, the five; that's exactly where it needs to be. This 10 miles is not, and let's see if I change that 10, 10 plus miles to 10 miles plus; let's see if that'll put it down here, cuz I, I don't know if it's looking at, I don't know if it's reading it weird, um, but let's go into this working sheet, and let's go right here, and we're going to do Control+H, and we'll do, oops, not this one, um, 10 miles plus; let's get that in there, and we're going to do 10, uh, miles plus. I, I don't know if that's actually going to work. Um, we will see. So let's go back to the pivot table; let's re-go to the data; let's refresh. Uh, no, it didn't, it didn't change it. Um, okay, so let's think about this. Maybe if we change it to like a letter, it might change down here. So start it with, uh, miles; that could work. Let's try it. Okay, it's already selected. Let's do the 10 plus miles. Okay. Okay, so let's do, um, uh, more than 10 miles, and we'll replace all. Let's get rid of this; let's go to the pivot and refresh. All right. Okay, so it's not perfect, but it works, um, and for what we're doing, I think we'll keep it how it is. So we have our second one, uh, and you know, there are different ways you can kind of change this one. Um, you know, on the last one, we did a ton of different stuff; we can do, just do commute distance, and we can say what do we want to say on this one? What is this? Oh, this is the count. Um, do we have to do, do we have to keep this one? No, there we go. I'm just going to do, um, just one and say commute distance, and let's add a title, chart title. We can make this one, um, let's say distance per customer. That's not 100% true because it's no or yes. That's, that's the important part of this; it's distance, um, average distance, uh, let's see; we'll just say customer commute. All right, and we'll keep it just like that. All right, perfect. I don't think, um, let me see; I don't think there's anything else we need to add on that one. All right. Now let's go right down here; we're going to create our very last one; we only have three. So you know, sometimes you'll have a ton; sometimes you'll have like one on each sheet, and you'll create multiple sheets, but, um, do Control+A, um, now we have our thing. Now, this one we're going to be looking at these age brackets that we were looking at that we created. Something that I do honestly a lot is is kind of bracket things into groups like this, and you know, for this, I'm just kind of made them

Up, but um, you know, it's good to know how to do this because I, I promise you, this one happens a lot. I use this one a ton. And then we just want to look at who purchased a bike, uh, so the same thing as we did before. So like, purchase a bike, count of the purchase, um, you know, pretty easy. So we just have to count of either no or yes for these age ranges. Um, and let's go to the insert; we'll go to recommendation. Um, I personally like a good line for this one. Um, so let's—this is already interesting—we could do something like this; that's nice. See this one versus this; it just adds a dot; it looks nice. We'll keep that one. Um, so just really quick, at a glance, really interesting: people under the age of 30 are not buying that many bikes. Um, age 30 to 54, uh, 31 to 54, buying a ton of bikes; uh, they, they buy more bikes or look at bikes more than anybody—really interesting. Um, but we'll make the dashboard a little bit—um—let's make these chart titles; we'll do vert—oops—the horizontal; we just call this age bracket. Um, and then we'll add a chart title. Um, again, you can add some extra stuff if you want to, um, but you don't need to. Uh, none of this other stuff we really need; I'm just kind of looking at the stuff we do need or do want. Uh, so what do we want to call this one? Let's call it customer age brackets. Um, and it's not perfect, but we'll keep it as is for comparison. Um, let me see if I can copy, um, or or use this, um, real quick. Instead of the age brackets, I'm going to get rid of this and use the age, and then let's use—um—let's insert recommendation; we'll use a line and we'll use this. So, this compared to this—just think of it like if a customer or consumer, or or not a customer, if somebody you're working with is trying to use this dashboard, understand this dashboard—this is going to be just—it's G to—I don't know, it might melt their brain; just makes no sense. It makes sense; it's just all over the place; it's really hard to make sense of this; it really is. I mean, you can kind of see a pattern going up around like the mid-30s, and then it trends downward, but it's hard to see. Um, it really is. So doing these, um, these brackets really helps, and you can even add, you know, adolescent, um, you know, 0 to 30 underneath it, and in fact, we may want to do that. Um, why not? Why not? Let's do that. Whoops. Um, so why don't, why don't we do that? Why don't we go back? I'm just gonna—I'm doing this on the fly—why don't we go back? Uh, what am I doing? Whoops. And this is all calculated, but let's do adolescent 0 to 30; let's do middle-aged 31 through 54, and then old 55 plus. Let's see if this breaks anything; I hope it doesn't. Um, and we'll go back to our pivot table; let's refresh the data. Uh, okay, it did mess with stuff. Okay, never mind, guys; that was a terrible idea; don't do that. Um, perfect. Uh, let's get rid of that; that was a terrible idea; don't do that. I'm glad we tested it out though. I like, I like to see if it was going to work. No, it messed with the, um, the order of things. Um, I, I intentionally named them adolescent, middle-age, and old because it's, it, it makes sense for the visualization. Um, but you know, if, if I change something and it messes with it, I'm not going to mess with it; it was just an idea on the fly, guys. Come on. All right, so let's start building out our dashboard now. Um, when we're building our dashboard, what I personally like to do is to have this pivot table sheet, and then I will copy them over, and later we'll hide these other sheets. Um, and I'll explain that in a little bit, but I like to have this, this one for us. So we're going to copy this, so I just click on it, hit Ctrl + C; we're going to paste it right over here. Uh, let's just make them small for now. That's—oh gosh, no, let's not do that. Oh, these look terrible. Okay, anyways. Um, let's copy this one over. Oops. Okay, what did I just do? Oh, I didn't copy this one. Whoops; it's not copying. Okay, we're going to go copy, hit paste, fantastic. Oops, guys, look away; this is, this is tough to watch; this is tough for me to watch; I'm the one doing it; it is tough for me to watch. All right, let's go to this last one. TR; I'm gonna try it again. All right, it worked this time. So now we have, um, our, our three visualizations; this is perfect. But now we actually want to create a dashboard. Now, how do you do that? How do you make it look nice? Uh, and then we're going to add some, you know, filters and stuff like that. How do we make it look nice? Um, what happened here? What changed? What do we do? Oh my goodness gracious. All right, let's copy this; let's paste this; let's get rid of this; I don't even know how that happened; I've never seen that before; that was wild. Uh, Excel is trying to destroy my whole video. I mean, I'm doing this for you, Excel. Good night. Okay, no problem at all. What we're going to do, and how you make this at least look nice, um, first off, we can get rid of these grid lines pretty easily, and I recommend when you do that, when you make a dashboard, just makes it look cleaner; makes it look like an actual dashboard. Um, let's go to view and grid lines, so we can get rid of these grid lines; it just makes it look nicer. Um, we're going to make—you know, we can choose any color here; I'm just going to get choose a color; I like this—and let's—we're basically creating like a header, right, if you're using like Tableau or something. Um, we're going to merge and center, so it takes every single cell that we have highlighted, creates it into one. Let's call this, um, bike sales. Uh, I have—I think I called it bike sales dashboard; let's just call it that. Um, you know, see what happens. Let's get that; let's make it white and make it much larger than it is. Okay, okay, that sure; let's do that. Doesn't look bad. Um, what is it doing there? We go. Uh, let's put that center, perfect. Um, it's not perfect, but we're going to use it. All right, so now we kind of want to organize these, and you know, everybody has their different way of doing it. Uh, I'm just going to start building it out myself and just see how it looks, uh, and then we'll go from there. I like this one there. Um, we can put this one—I, this one's a kind of a longer one, so I'll probably put it at the bottom. Let's see how it looks. Um, but we'll put this one right here; try to line it up. Geez, let's, let's zoom in a little bit; let's try to line this up; see what it looks like; let's extend it to the end; that doesn't look too bad. Uh, needs to move up just a hair, and I'll show you how to kind of, of align these in a second, but, um, that looks not bad, and we'll kind of try to align these as well. Let me zoom out and extend this the length of this just to make it look nice. Um, you know, now what you can do, and you know, this is something that's pretty simple, is you can get both of these, and we're going to go to shape format, and we can just align these; it's really nice to align, especially if like the top, um, it maybe like the left to right, but like, we're going to align these to the top, and they just kind of align themselves on the very top. Now these look much better. This one is a larger dashboard or a larger visualization, so I'm going to keep it how it is. Um, and I'm going to keep this one how it is, so it is going to be a little bit smaller, as you can tell, and then we'll have this one. Um, and I'm going to do that. I, this is going to bother me if I don't align these, so let me do this; this a shape format, align to the right, and it's not exactly what I wanted to happen because—oh jeez, what am I doing?—this not exactly what I wanted to happen; I actually wanted this one to align, uh, this one to align with this one and did the opposite. Um, so let me just scoot this back. All right, visually it looks fine, but that's how you do it if you want to do it. I, I, I, if you have multiple of them like this, you can make it look, look bad. So we have our dashboards; this is already looking really good. I, I like how this looks; colors are coordinated; it, we have a kind of a theme throughout, um, and it looks nice. I actually, I actually kind of want to change this one, um, to—um—let's see, maybe if I did like that, it look nicer than all of them. Yeah, this does look nicer. Um, it doesn't change much either, guys. I'm, should I do it? All right, we're going for it; we're changing the design on the fly; should I do it for all of them? Let's see; it doesn't fit; doesn't fit. Um, all right, guys, just ignore what I'm doing; don't do any of this; I'm just messing around at this point. So this is really great to have; it really is. And what we want to do is there are other elements; there are other things that people would like to feel a, to filter by and be able to look at, but it's not in this visualization. Um, to be more specific, one field that's could be really interesting is married versus single: are single people buying more, or, um, married people buying more? You know, it, it'd be nice to filter on it. So we're going to click on, uh, any of these actually, and we're going to go up to PivotChart Analyze, and we'll click Insert Slicer. Now we can choose which ones we want to be able to filter on all at the same time or one at a time. I'm just going to do the first one by itself, and then I'll show you how to do other ones. Um, but this one is the marital status; so this is the married, single, the one we were just looking at, and we can drag this right over here; get in a little bit. All right, and we don't need all that space, so we're going to boop, boop, boop, boop all the way up. Now, while we're doing this, um, it only because we selected this, uh, this visualization, it only is working on that one right now. We, of course, wanted to apply to all of them; is not hard to do. All we're going to do is we're going to click on—we're going to make sure we're clicking on this—we're going to go up to slicer; we're going to hit Report Connections. Um, and if you remember, we have this, um, this pivot table that we're working with, um, and this is where all of our pivots are coming from, so we're going to actually apply it to all of them. This is our sheet, um, and this is the name of the pivot table. Now, again, we created that fourth one; we're not using it, but we're going to apply it to all of them. So now when we click on it, it's going to apply to all of them. So, at a quick glance, let's see what single people are doing. Um, interesting, interesting. Um, you know, when I'm looking at the just these numbers right here, married people, these individuals are making a lot more—like eight, um, sometimes eight to like 10,000 more on average than their single counterpart. Um, you know, again, that's a rough estimate, but it's, it's interesting. So now what we can do is we're going to create more of these. So we're going to go to, uh, PivotChart Analyze; we're going to go to slicer. Now we already did marital status, but what if we want to look at things like, uh, region and maybe something like their education? So let's bring up both of those and look. Now two of them come up. So let's add the region right here; bring that in just a little bit; see if we can match it; nailed it. All right. Now we're going to, to put that up; we'll bring this one down just like this; bring it over; see if I can match it again; come on; almost nailed it; I don't know if I nailed it, but it's close. All right; kind of bring this up a little bit; bring this up; and we have to do the exact same thing that we did with this one because, right now, again, it only applies to that one, um, chart. So what we want to do is we want to go to slicer, report connections, add it to all of them. Okay; do the same thing with education, or connections; bada bing, bada boom. We are looking good. And now, uh, let's get rid of all of them; this just going to be everybody. So now we can kind of slice and dice and choose what we want; we want to look at people who have a bachelor's degree who live in Europe and are single, and this is the information that we have on those people. So now we can narrow it down by certain demographics even further and look at this key information. So we may not, you know, look at counts and averages of these things, but we're able to filter on them, uh, and that's really great to know. So bachelor's degrees, on average, are making 60s, 70,000. Um, let's look at, um, let's look at graduate degrees. Okay, a little more. Um, but you know, again, I'm just looking at random stuff. Um, but you can mess around with this; take a look at some stuff. Um, this to me—I want to make this color darker; I feel like it look nicer, darker. There we go. Oh yeah, that's way better. This to me is—it's a good dashboard, right? You have key information that you're looking at; nice visualizations; it's color coordinated; you have these slicers on the side. Um, to me, this is a fantastic, just simple dashboard, and there are so many other things that you can do with this data, and you can make unique, and you can add your own spin on it, and I highly recommend that you do that; push yourself; go past what we just did today and add your own stuff, and, and use this, and then you can add this to your portfolio website and show this off and show people that you know how to use Excel, which is a fantastic thing to know how to use and show off. So, with that being said, I hope that this project was helpful; I hope that you learned something along the way; I know I did. Um, I was learning things as we were going, and I hope that you didn't mind that I took some detours along the way, um, for your amusement as well as my learning. Uh, so with that being said, thank you so much for joining me; I really appreciate it; I hope you have a good day, and [Music] goodbye.

What's going on, everybody? Welcome back to another video. Today we are starting our Tableau tutorial series. [Music] Now this series is for absolute beginners, so if you have never used Tableau before, you are in the perfect place. I'm going to take you all the way from the very beginning of installing it and just understanding what Tableau is and how you can use it all the way to creating dashboards and sharing it. Now, personally, I hate those videos that are like 3 hours long and they just expect you to go through it. Uh, I like to break my videos up in the chunk, so if you have ever done my sequel tutorials, you'll know that I like to break things up, so it gives you time to try them out and do them yourself, and then you can move on to the next video. So I'm going to be breaking this up into five separate videos, but in this video, I'm going to show you how to install Tableau for free; I'm going to show you the user interface; we're going to download a data set that you can find on Kaggle, and then we will build our first visualization together. With that being said, let's jump over my screen, and we'll get started. All right, so the very first thing that we need to do is you need to actually download Tableau. So we're not going to be using Tableau; we're going to be using a free version called Tableau Public. It has a lot of the same features, features except, of course, it's not, uh, every single feature that regular Tableau has, but it is absolutely perfect for learning it and for using it, and, and you can even build, um, you know, dashboards and share those for your portfolio. Um, I'm going to put this link in the description, so you can just go and click on that, and, and all you have to do is input your email right here. We're going to click download the app, um, and then it should start to download, and then you can save that, and then you're going to open this up. Now I'm going to open it up; I know what it's going to do; I already have it downloaded, um, but it should open up and look hopefully like what you're seeing on my screen in just a second. Let's see what it does. Um, I hope you can see this, but it says Tableau Public. Um, it says I already have it set up, but you're going to click install and go through all that, um, all that setup stuff. Uh, so I'm going to exit out of here, but I'm going to go over here and type in Tableau Public, uh, and it's 2021.3; that's the current version that they have out if you're doing this in the f. Um, so you should be able to pull this up right here now. Um, I'm going to go and get our data set that we're going to be using, and I'm going to show you how to get that as well, and then we will actually jump into Tableau and start, uh, using it. So let's go over here; I'm going to get a data set from Kaggle. I wanted something pretty generic, uh, to show you in future videos; I'm going to show you some special, or not special, but just different visualizations that you might use, um, and we'll get different data sets for those because, of course, not one data set covers all these other types of visualizations. So, um, we're starting off pretty simple right here; we're going to be getting one called video game sales, um, and we can take a really quick look at it. Um, here are some of the fields that you're going to be having, uh, like rank, name, platform, the year, genre, and then some sales data, and this is what it actually looks like; it's called VG Sales, so video game sales; it's in a CSV, and, um, you know, here are the fields, and we have our data, and all we are going to do is we're going to download that, and I will save it. Now, when you download it, it's going to be saved into a zip file, so we need to go to our downloads. Uh, let's refresh this; here's our archive; we need to go in here; you can just copy it and paste it right back into here, um, and just so you know that is a, a CSV, so be aware of that. So what we want to do is we want to come in here. Now, since it is a CSV, this is not—we're not going to be using Microsoft Excel; we're going to be using the text file, so we'll come in here; we'll take VG Sales. Now, uh, one thing I want to do before I do that is I'm going to rename mine VGC Sales1. Um, I've already prepared for this, and so I already have that in there, um, but so I want to make a distinct one for myself; you do not have to do that. So we'll come back here, um, and then we're going to do text file, VG Sales; we're going to open that up, and when it pulls up right here, um, you can bring in other tables, and then you can start to join them together and create those relationships; we are not going to be doing that in this video; we'll do that in a separate one, um, as for, you know, just getting started, you know, we're not going to be using that, but you can see, um, some of these things, or some of these fields, and if you notice, they, they, um, they're either ABC or they're a number, so it starts to categorize.

What this field type is, so is it a string? Is it numeric? It starts to automatically do that, and that's all done within Tableau. And so it just kind of reads it, and that's what it does.

Um, what we are going to do is I'm going to click right down here; it's called "Go to Worksheet." The worksheets are where you're going to actually start being able to build your visualizations, your charts, your graphs, all these things. Um, and so, you know, we have this in here now, and so we're just going to click right here on "Go to Worksheet." As you can see here is VG_Sales1. You will not have the underscore one if you did not add that like I did. Uh, but right down here you can see all the fields that we just imported from that data set, and they even created one right here for us. Uh, they just generated that field based on the file, so it's a count of all the rows really.

So what I'm going to do is I'm just going to walk you through, uh, basically what we're looking at, some of the things that we're going to be using today. There will be things that I don't talk about, but I'm going to highlight those in in in future videos when we start using those or going over them. Um, and so let's just start with the most obvious one; it's way over here. I'm sure you saw it when we, uh, this first came up on the screen because it has all these different charts and visualizations and graphs and, uh, these will become available as you start dragging and dropping our data into this sheet. And so if I go right here, it says "four Scatter Plots, try zero or more Dimensions, two to four measures." So what our dimensions are are right here; what our measures are are right down here. And so typically, uh, things like like you say genre or names or or strings like that are going to be these uh dimensions, and then a lot of lot of times the numeric is going to be are going to be measures.

Next, what I want to show you is right here. So you can take something like Global Sales and you can drag it right here into your rows, and then it takes your rows, and so it automatically created a sum of Global Sales. Now if we take that away and let's say we drag it right here, it's going to give us a column. Now you can also do it right up here; you don't have to um drag it on screen. You can also just add it to the column or the row. That's typically what I do; it's just more intuitive to me. Um, or you can drop it in this section right here, and it does its best to assign it some type of um some type of visualization. And so that's what it always is trying to do; it is trying to say, "Okay, this is what you're trying to do; let me try to get the best visualization for the data that you're giving me."

Now, while we are here, um, it went down here into Marks, and Marks is a very important area; it's where you can add color, size, text, detail, and Tool Tip. And I'm not going to go into what all those are because I'm just going to show you. So let's start pulling some fields in here and creating a visualization, and then I'm going to show you how all of that works, including filters as well. So the first thing that we are going to look at is Global Sales, and let's put that in the rows, and then I'm going to take Year and I'm going to make that at the column. And this is basically exactly what, uh, I wanted to do. Now, as of right now, it has only the year, and it's looking at Global Sales for everything, but we want to break that out a little bit better. I want to break it out by, let's do Genre, so different genre of games. Now if I add that right here to this columns, it is going to break it up by year and genre. If I add it right here, is going to break it out by the year of course, but then in each individual row has the different genre; that's not what we want. We want to keep this type of line graph, uh, and what we're going to do is we're going to add it to Marks. And you can't really see it based off of these colors, but they're all different, so we have Action genre, we have the Sports genre, Racing, uh, Role Playing, all these different genres within it. Now, can get rid of that because we don't need it anymore, uh, and this is where these um these Marks really come in handy because you can start basically doing what you want with them. So for the Genre, I want to be able to see all these different genres with different colors; to me that just makes the most sense. So I'm going to put Color right here, and automatically it assigns every single genre its own color and gives us this Legend right over here. And so it's really easy to see; well, when you have smaller numbers is much easier, but I know that red is Sports, and I can go right here and find red, and that is Sports. So it makes it a lot easier than when it is all the same color blue.

So what you can do after that is you can also add things like uh a label to it. So if we take Label and we or we take Genre, put Label, you can click right here, and you can get rid of the labels that you have, and you can see them right down here, or you can also change uh the font. So if you want to make it orange or or whatever color, you can do all those same things, and you can also do things like changing where you see these things. So for Action, you're going to see it a ton because for each year Action is is at the is on the higher end, and so you're seeing those in those mins and Maxes. You can also do it for a selected area. So if I come in here and I select it, it's then going to show me what those are. So Label is really really uh useful, really help; let me get rid of that really quick. U You can also do it where the lines end, so Line Ends is at the beginning and the end, and you can also take that away or put that back on. So labels are really important; labels aren't very helpful when you're doing, at least I don't find that it's super helpful when you're doing things like Genre, so when you're doing your Dimensions. So I'm going to get rid of that, and I'm actually going to bring our Global Sales over here, and let's label that. And right now I think it's labeling the uh line ends; we want to do the Min and Max. Now if we do Min and Max on the table, it's just going to give us the Max and the Min, which is zero and then 139.0. It's a little bit more useful if we do it for each line; this at least gives us some context. I probably wouldn't do this in an actual visualization, but to give you some um understanding just how it works. So now I know that um right over here the men and the max or the men, sorry, the max for these for Action and for Sports is right around 138, 139, so it's pretty easy to see. Um, and you can again go in here, and you can remove the max or remove the mins, whichever one you feels best; you'll probably keep the maximums in there for each category. And so this is really quickly becoming uh a pretty usable visualization, and that's not the only label that you can add; we still are using Year over here, so we can always drop Year in there as well. We'll create a label, and so now we have, let's see, for this one is a Puzzle genre, so we also have the year that it had the maximum uh sales. And so, you know, just some things that you can do; you don't have to add that.

Now let's go up here, and we're going to take a look at Filters because Filters are really important. You know, if you are making this for a client or you're making this for somebody, you want them to be able to filter down uh to very specific information that they want to see. So let's take uh the Platform; lots of different platforms, um, as you can see, you know, PS4, Xbox, um, if you're familiar with these, we'll click all of these, um, and we'll click Okay. So now this is an option as a filter, and all we're going to do is we're going to click on this Arrow right here, and we're going to say "Show Filter." Now right now all all of them are selected, so every single one is being taken into account for this visualization, but let's say we come down here and we say, "Okay, I don't want to see sales for any of these PS, the original PlayStation 2, 3, or 4," so I'm going to get rid of this one, this one, this one, and this one, and you could immediately see the the changes that were happening. So now none of the numbers, none of those sales are being accounted for and and being added to the sum of Global Sales right here at all. So that is just how a filter can work, and you can also do that, and you can get rid of all of them, and you can go in and actually just pick very specific sales. So if you only want to see the PlayStation sales, you can go in there and do that as well. So really really handy; Filters are things that you'll at least want to have as an option for most of your your visualizations; at least that's what I found, especially when you're doing client-facing work. They like to uh get in there and mess around and look at different look at it in different ways. And so that's one that I I think is is really useful.

The very last thing that we want to do is we want to actually add this to a dashboard. Now let's say we add, come right down here, and we add a new worksheet, and actually we might change one more thing on that last one, but we'll just make a really simple one; um, we'll just give it Genre and we'll give it Global Sales as the rows, um, and this Nifty button right up here, which is a sorting button. So I'm going to sort like that; I'm going to add the Genre in just as we did; I'll give it different colors; perfect. Now we have two really quick different visualizations. Right, what I want to do is just show you how to combine those because what you are going to do is you're going to actually come in here, and you're going to do "New Dashboard"; that's what this button is right here. Now when we come in here, the size is extremely small; it's very easy to fix that; all we're going to do is click right here; we're going to go to this range or this drop down, and we're going to click "Automatic." So now it is a much larger size for us to actually drop our visualizations into, uh, and let's put Sheet one and we'll put uh let's put it up top. So now it looks a little bit like this, uh, not perfect, but again, if I wanted to make this look a lot better, I definitely would, and then you can go over here, and you can rename these things; you can also do that back when we were in our actual worksheets, but you can also do it here as well and then start um, you know, customizing and building it out. That's not what this video is for; that is the last video. We're going to build an entire dashboard; it'll be kind of like a small project; you can put that in your portfolio. Um, if you have gotten this far and you want to jump straight into it and you don't want to wait for these other videos to come out or you don't you just want to jump straight into creating an entire portfolio project, I have an entire portfolio project series that covers SQL, Python, and Tableau. And so go check out that series; I have one video dedicated to Tableau; it's like 45 minutes or an hour long, and it covers a lot of the things that we're going to hear in here as well as a few other things. But I appreciate you checking out this video. In future videos, we're going going over things like creating bins, calculated fields, doing joins, and then creating a final project and putting it all together. So thank you so much for joining me; I really appreciate it. If you like this video, be sure to like and subscribe below, and I will see you in the next [Music] video.

What's going on everybody? Welcome back to the Tableau tutorial series. In this video, we're going to be going over bins and calculated [Music] Fields. All right, so let's jump right into it. The first thing that we're going to look at are bins, and bins are basically just groupings or ranges of numerical values. So we cannot create bins uh for Genre, Name, Platform, or anything like that; we have to do something with this sign right here, which means that it is a numeric; so Year or all the sales data or this ranking data. And we're going to use what we worked on in our very first tutorial. And so what we're going to be using to kind of demonstrate how bins work is this Year right down here. So right now we have a range of 1993 all the way up to 2018, and we're going to create some bins to group and create ranges for these years. And it's pretty simple; all we're going to do is we're going to come right over here to Year and this little drop down on the side, and we're going to go down to "Create" and go down to "Bin." Now it's going to say the size of Bin, and it's going to give you a recommendation based off of the information that is already provided, the Min and the max, the ranges of these values. You know, you don't have to do this, but usually um it it does give some good estimation on what you might be considering. If you were thinking, "Hey, maybe do a bit of like 20," and they're recommending two, think about why they might be doing that. Uh, we're going to change ours to five, and you can always change what this field is going to be; I'm just going to give it an old exclamation point just to um really spice things up here. So we're going to click Okay, and as you can see, it adds it right up here; it is no longer um it is no longer a numeric; now it is a categorical. So now it's this is no longer just uh 1, 2, 3, 4, 5; its ranges, its groups. And we're going to get rid of this Year really quick; actually, let's keep it up there for a second, uh, see what happens, but we're going to bring this up, and we'll get rid of this Year, and this is is what kind of it spits out for us. Now I did look at the data um when I was prepping for this; there are some nulls in the Years, um, and so all we're going to do for this is we're just going to go like this, and we're going to exclude the nulls; probably not something you should be doing uh if you're doing this for work, but this is for demonstration purposes, so we can do whatever we want. But as you can see, we now have these ranges, so this range starts at 1990, and it includes 1990 all the way up to 1994, and then it's 1995 to 1999, and so just really quickly we can tell that the years 2000 to 2004 were a huge huge huge uh season or group of of years for game sales. So these are the Global Sales for for these video games, and so it is really helpful; it's very useful. Um, you can do this on a lot of different information; we could do this on the sales data; you can do this on age; you can do it on years like we did, and it can be very very useful. And so uh really quickly, that is how bins work; I would say it's pretty straightforward.

Now this is a perfect time to segue into the next part of the video, which is Calculated Fields. Uh, right over here on this left-hand side, we see that the Global Sales, which are in millions, goes all the way up to 900 million and created these beautiful bins right down here, but let's look at within these from 1999 to 2015; let's see which of these has the highest percentage. Of course, it's going to be this one, but we can do something called a quick table calculation. U We'll create a our own calculation later; I'll show you how to do that, but we're going to do a quick table calculation, and we're going to do the percent of total. And so now we have these bins, and instead of just seeing the total amount of sales that they had, we see the actual percentages based off these year ranges, which is is really useful; something that you could absolutely put uh in some real work that you do for a client. Now really quick just to show you something that you can do; if you click Control and you drag this over here, you can actually save that calculation. So we can say "Percentage of Global Sales," and that actually saves it as uh, you know, a measure for us. So that was a quick calculation, but let's look how to actually create a calculated field. So if we do this right here, here what is going to come up is just the Global Sales, and you can do a lot of what you would basically do in Excel; multiplication, division, subtraction, a few other things, but we're going to keep it super super simple today. All I'm going to do is I'm going to take Global Sales, and I'm going to subtract; I'm going to do an open bracket, and I'm going to say EU Sales, and it auto completes for me. I'm going to click Okay, and it created Calculation 2. I'm going to come in here, and I'm just going to say "Global Sales - EU Sales," and let's drag this over; these are different um one's percentage, one is in terms of sum. And so I'm just going to bring this in right here, and so now we are comparing against the same thing. And if we look at the Global Sales, we have probably right around 950 million-ish in this 2000 to 2004 bin, and for Global Sales minus the EU Sales, we're looking at, you know, 650 million, so there is a noticeable difference. And this is just one of the ways that you can use uh calculated fields to actually just show the difference between two numbers, or you can do more advanced calculations depending on the data that you actually have. So that's it for this video; I hope you learned a little bit more about bins and calculated fields. In the next video, we're going to be looking at a ton of different visualizations and graphs and charts and just exploring what options are really are out there for visualizing our data. Thank you guys so much for joining me; I really appreciate it. If you like this video, be sure to like and subscribe below, and I will see you in the next [Music] video.

What's going on everybody? Welcome back to the Tableau tutorial series. In this video, we're going to be looking at lots of different visualizations, including the Scatter Plot and D [Music] Maps. Now, before we jump into the tutorial, I have some very exciting news. In just two days, on October 7th, I'm going to be partnering with Alter X to host a webinar. This webinar is completely for data analysts who are wanting to change careers to become a data analyst. Now you did hear that right; I will be the host of the event, but we will be bringing on guests as well who are industry experts who actually changed careers to become data analysts, much like myself. They'll be sharing their stories of how they actually transitioned careers along with the tools that they found extremely useful and helpful to make that switch, and they'll be giving lots of advice along the way. So if you are somebody who is wanting to change careers to become a data analyst or just wanting to learn about data analytics, this is an absolute fantastic place to learn a lot more about that. I will leave a link in the description, so be sure to go and sign up for that. Again, I'm going to be there, so it should be really fun. Without further ado, let's jump onto my screen and start the tutorial. Now we are about to look at a ton of different visualizations; uh, over here you can see just an array of them, but not all of them are…

Ones that I actually think are useful, or ones that I would actually recommend using. And so I'm going to take you through some of the ones that I absolutely think are worth learning and using and trying out. Uh, and I'm just going to kind of just show you how I might use them, how they might look, how you can navigate them a little bit.

Now, before we do that, we do need to go download one data set. It's this Starbucks location worldwide. Yes, we're going to do a little bit of longitude latitude here. And all we have to do is click this downloads button, and it will download. We're going to do that into downloads; we'll save that. Uh, yeah, I've already done that, but you know, I'm doing this with you guys, I'm doing it for you, so let's go to our downloads now. We have here; we want to come in here; we're going to copy it, or um, you can cut it, uh, and then we're going to paste it here. Yeah, replace it, perfect. And now we have it ready to go. We'll come in here; let's do a new sheet, and I already have it in there, but uh, I'm just going to show you what I would do: a do new data source, we'll do text file, we'll do directory, and we will open it. And let's see what data we have in here before we actually begin. Uh, just super quickly, we have the brand, so whatever company has it, and then a bunch of um location information: street address, City, the state. This is all in the United States, so that's basically it. And what we are going to do is we're going to go over to this sheet three, and we have this directory 2; that's the one I just pulled in, exact same thing as directory.

But so the first visualization that we are going to look at is a bar and line graph. So what we're going to take is the year right here and take these Global sales and these NA sales, and we're going to be doing this one right here. So this has a combination of two separate uh types of visualizations. So sometimes you just have line, sometimes you just have these uh these bar graphs or these bar charts, uh, and we're combining the two, and it's very nice. I like how this looks. Now, if you notice, if I put this NA sales behind it, now it kind of cuts off. So now this Global sales Z in front; we're going to, you know, put that back. I just wanted to show you that. Uh, right here, there's all some of global sales, some of NA sales. So if we go into this, all we click this drop down, we can change it to a line. Um, we can change it basically whatever we want. I just hit Ctrl+Z to reverse that, but what we can do is we can go in here, and we can change this color. And let's see if we can just make it red. Is that possible? See what I did? I made it orange; that works for me. Um, just something to stick out a little bit more. Choose whatever color you want, and this is a really nice visualization. This is one that I have used in the past. We're looking at Global sales versus the NA sales, and so it's very easy to see the distinction between the two and how one was doing a specific year versus how the other one was doing in that same year. And so I really like this. If you want to do something uh like keeping it consistent, you can do two bars. I don't really like this one as much, um, and you can, again, you can really change it up. Um, there's lots of different ones that you can do. Again, I prefer the line, but you know, do whatever you think is best. I'm going to change it back because this is not how I want to keep it, but there you go. So that is the first one that we are going to look at. Let's move on to the second one, and we actually will be using our our Starbucks data here.

Now, when you bring in data that has um any type of map or or um address or postal code or things like that, or or country, it's typically going to create this latitude and longitude; it's going to generate that. Now, what we want to do is bring this longitude right up here and this latitude right there. And if you do the show me right now, it's giving us this, but what we want to do is add what we're looking for. So what will we actually be trying to search for on this map? You can do anything from like a postal code, um, and it will drag us right here. Let's come over to this; this allows us to kind of scroll around a little bit. Um, we're going to mess around with this one for just a little bit, and let me see if I can. That's nice; that might be too big. Let me back up one. So, at least in the Continental US, a little bit down here, this these are the postal codes. So right now we're looking at post codes, and there are a lot that you can do with this. Um, really, color will make almost no difference; it just becomes this mess. So you don't typically want to do something like that, at least not for this. Let's go to size, and if we make it really small, you can kind of see these groupings, these pairings, um, typically of like larger cities or major major metropolitan areas. And so you can do this, and it's and it's really really easy. I don't recommend uh labeling this; I don't even know if it'll do it. Um, it would be an absolute mess to try to label all these postcodes, but let's bring this out, and let's bring these State and provinces in now. Right now we have these little tiny tiny uh dots on here, and I think what we want to do is not increase the size, but over here we want to actually do this and make it a map. So now it's going to fill in all the States. We can, you know, why not, we'll add some color here, um, but we can; it has a numbered; I didn't think they were numbered. Um, oh, that's interesting; I haven't seen; I didn't look at that before. I was just found that interesting, but now we can see what uh what states Starbucks is in, and as you can see, they're in all 50 states, but it's something interesting to um look at, to think about. Now, if we go right up here, we can again choose a different type, and we're going to go to the density. Now, right now it's just doing a density on the uh the state; we're going to get rid of that; we're going to bring back postal code. I'm just switching it up on you a little bit, and you can do it as small or as big as you'd like. Um, you know, I like to do somewhere in the middle, um, probably right May, right about there is fine. Um, I don't think it's going to make sense to really add any color here again; all these poster codes are different, so it's just going to be complete mishmash, but uh, this is kind of how you can use a density map, and you can do this with uh countries; you can do this with postal codes; you can do this with any type of kind of like address or location-based data. So that is how you can use a map. Again, there's lots of different ways to use a map, and so I'm not going to show you every single way, but in a really brief way, this is how you can use a map to actually visualize your data that does have location uh based information in it.

So let's go over to sheet three, uh, and this data that we have over here; it just allows for a lot of different types of visualizations. So we're going to use this one, um, and there are lots of other ones that you might see out there, like this one right here. Uh, we obviously wouldn't be using this; we might do something like this, change the label, um, and maybe add why have both of these in here? Um, let's get rid of this; oops, that's not what I meant; let's actually add that; let's do the sum of global sales, and we'll just make that into a label as well. So what you can do with these and and how you're able to use them and visualize them. Again, these are not; you'll see these often, but these are not often ones that I would recommend you use. That's very similar to these packed bubbles. Um, you can add these Global sales in here again, add the label; it just uh it sometimes is not as straightforward, the information that is trying to tell you; right, you kind of have to search for it a little bit; you kind of have to look around, um, but you can find some good visualizations in here for very specific types of data. And so these are just ones to consider. Uh, one that you'll see all the time is uh this guy right here, and uh let me see if I can expand this a little bit. This is very very small. Um, let's see; I have the; I just want Global sales and let label that the size. I how do I expand this? Haven't done this in a while. Let me just expand this; I don't use pie charts. Up, what is happening? This is an incredibly large pie chart. Oh my gosh, I am making this; this is becoming a problem. There we go, uh, and what I actually wanted to do was label the uh genre as well as I've been doing in all the other ones, and we'll label this. Now, look, whether you are a fan of pie charts or not, you have to understand that people use them. Uh, some people just like how they look, and for certain data, it can do well for things that have a lot of different um groupings or categories; it usually isn't super great, uh, but it does give you some type of order of things, give you a quick glance, and people use them, right? So let's not pretend like it's like the the the Hideous stepchild. All right, people use it; people have it in their dashboards and their visualizations all over, so it's best to just know what they look like, know how to do them, know um how to use them best. Again, I'm not a super huge huge fan of it myself; I've used it once or twice, but one to look out for. And again, you can come over to here and use is called a box and a whisker plot. Um, it's good for these large um distributions; you know, this is like the median, upper upper, lower lower. I don't use these a lot, but I know a lot of people who love them. Something to just look at and consider; mess around with it a little bit; it's pretty; I think straightforward, and it does give you some good insight into your data if you know how to use it.

Now, there is one last one that I want to show you. I'm just going to create it on a new sheet; make it easy. Uh, we'll do year here; we'll do some of; let's do NA sales; why not? And we are going to make this like this. Now, it's very similar to a line chart, but when we break it out by the genre and we add some color, you know, it's just a different way to visualize this information. You can uh, you know, potentially add some stuff in here like some labels if you uh want to, depending on how it looks for you, but this is just another way to visualize the data. So wanting to give you guys some options, wanting to give you some things that you might want to look at if you haven't already used these before. These are ones; all every single one that I've showed you are ones that I've at least used once. Um, this one I've maybe have literally only used once, but the first ones that I showed you, the ones I pointed out as the ones that I really wanted you to know are great visualizations to learn how to use and learn how to make useful for the data that you have. With that being said, that is all that we are looking at in this video. Again, I tried to keep it super easy; just wanted to show you some different visualizations, the data that you can use to get those visualizations, and just some other options in case you wanted to get a little bit uh spontaneous, a little bit out there, a little bit funky uh to show your boss or something like that. Thank you guys so much for watching; I really appreciate it. If you like this video, be sure to like and subscribe below, and I will see you in the next [Music] video.

What's going on everybody? Welcome back to another video. Today we're looking at JoCo and [Music] Tableau. Now, before we get into the tutorial, I want to give a huge shout out to today's sponsor, and that is Udemy. They were having a massive Black Friday sale, and so everything is about 85% off. So if you've been looking at a course, now is the time to buy it. If you are looking at learning and taking an actual full Tableau course, there are fantastic ones on Udemy that I have taken myself. So be sure to go and check out Udemy while they're having this huge sale. I will include a link in the description if you want to check them out. Now let's get into the tutorial. All right, let's get started. And first, we're going to start off in Excel. I'm going to kind of walk you through the data that we're working with, and then we're going to put it into Tableau, and I'm going to show you how to do all those joins in Tableau. So the first table that we have is this demographics table. We have employee ID, name of employee, employee age, and employee gender. Now, look right here because this will be important uh going forward. In the demographics table, we have 10 uh IND individuals, and they each have an employee ID. Now, when we go to the job title, we have our employee ID, employee name, and the job title, but this one is missing; Ryan Howard is missing his employee ID. And then the very last one, there are only seven employee IDs and no names, um, and so we're going to use all of that, and I'm going to show you how to actually do the joins in Tableau. Tableau does a really fantastic job of visualizing for you, so it takes a lot of the guesswork out. Um, I am going to include a link to my joins video in SQL because these two are very closely connected, and and if you understand how the joins work in in SQL, you'll understand how the joins work in Tableau; it's almost the exact same thing. So with that being said, let's jump over to Tableau. So I'm going to pull this up, go right over here, and now we have uh where we can connect to our data. And so we're going to click Microsoft Excel. I'm going to scroll down here to T joins file. I'm going to open this up, and I have it open, so I can't use it, so let me get rid of that, and let's open it again. Perfect. So now what we're going to do, and I'm going to show you how to actually open up the joins um in a second, but what you need to understand is when you first come here, Tableau doesn't automatically allow you to to use the joins; they use something called relationships. And there are joins on the back end, but they call it relationships because they are inferring all of these things; they're trying to go in and make that inference for you, so it takes a lot of the work off of you. And most of the time that works, and and you know, you just plug these two things in here like a demographics and the job title, and it is going to, you know, help you build those what they call relationships. And you can click on this and learn how the relationships differ from joins. Again, there's not a huge difference, but it's not as customizable, and you can't as easily do left joins or full joins or all these things that we're about to look at. So uh I'm going to take this one off, and what we're going to do to actually be able to look at the joins and and choose what joins we want to use is we're going to do this drop down; we're going to click open. And so now we are in a place where we can actually create the joins, uh, and again, it's just much more customizable. And so um back when I was using Tableau regularly, I would use the relationships when it was pretty simple and straightforward, cuz they almost always got it right, but uh, you know, the joins, it it just makes more sense in the way it visualizes it for me. So most of the time I'd be using the joins. So let's pull over this job title right here, and it's going to make this connection. Now, before, if you remember just about, you know, 30 seconds ago when it connected them, it was just a line, and and so it gave us this option down here to kind of edit the relationship, but now it's giving us this visualization. And so let's click on really quick, and what is going to come up is the different types of joins that you can do. You can do an inner join, a left join, a right join, and a full outer join, and then you can actually choose the different uh data sources and how you're connecting them. So again, um, I'm going to walk through a little bit of this, but I think the SQL video that I did on this shows it so well, um, I would just highly recommend using that, um, and I recommend learning SQL too, so you know, two birds, one stem. So I'm going to get into each of the joins, how they work, what data is going to be displayed, um, and these visualizations are really going to be helpful, and I think that it's it's just nice that they have it because it's a little reminder, okay, um, you know, this is what this join is, or this is what that join is. So super super simple. So right now we have the demographics table, and we have the job title table. And so what it's doing right now, and let's get rid of this, what it's doing right now is it's doing an inner join. And so it's pulling everything that overlaps if it matches on the employee ID and the employee ID. And so right now you only see 1 through 9, but if you remember in the demographics table, we had uh 1,000 all the way through 10. So where is that 10th one? Well, the 10th one is not there, and that is because in this job title employee ID, it only went up to 9, and then Ryan Howard just didn't have an employee ID in there for whatever reason. So that data is going to be missing. Now, when you are using actual data sets, very large data sets, which we will use in the next video when we walk through an entire project, um, when you use large data sets, this can be the difference between clean data and very wrong data, and and visualizing it correctly and showing completely wrong numbers. And so you really need to be sure you understand how your data works together when you're doing these joins. So how can we fix this? How can we um make it to where we can see all of the data? Well, right now we're only making it to where if the employee ID is equal to the employee ID, so we only are going to see through 9, 3, 1009; we're never going to see Ryan. So there are two different types of joins that we could do to make it see it, and then there's something else that we can join on to where we can see that data. The first one that we can look at is the right uh join, and what this does is it's going to take everything that is the same, but also everything from this job title table, regardless of if it has a match in the demographics table. So it's pretty, you know, this visualization does it all; it's going to show everything in the right table, regardless, and it's only going to show things from this table if there's a match. So let's try this one, and we should see Ryan Howard in the job title table. So let's click on it, and if we scroll down, there's going to be N, N, N, N, N until we get to over here where we now have have the data that we had in that actual table, but again, this wasn't a match, and so we weren't able to see that data. So this gives us a way to where we can see all of it, um, all everything from that right table, this job title table. Now we're going to click on the

Full outer now the full outer is going to take everything from both, regardless of if there is a match at all. And so right here, you're going to see Ryan Howard and Ryan Howard. Now, why are there two different rows for it? Well, because in the demographics table there was an employee ID, so we're seeing the employee ID Ryan Howard, his age, and his gender. And over here, there was no match, right? But in the job title table, again, this one didn't have an employee ID, and so we we are going to be able to see this data. But over here, it has no match, and so that's why it's showing us two different rows; is because there was no connection, there was no match there. That's what a full outer join is going to do.

Now, just for uh the purposes of seeing what this one does as well, we have the left-and table um and now we are able to see the 110 or or 110 that we didn't see before um and it's putting in nulls over here because there's no match. So that's that is um what we have so far.

Now, like I said just a second ago, there is a way that we can do this without using the employee IDs. We're allowed to use a a different join clause. Now there is the name of the employee in both of them; this one is called name of employee, and in the job title it's called employee name. They don't have to have the same column name in order to join it; you can do whatever you want. So I am going to get rid of this one, and now we are only tying it on the employee name, and let's do an inner join, and it should be basically everything um except the only piece of data that wasn't filled in, which is that 10,000 over on the job title table. And so this way was a slightly different, maybe a less thought-of way, because normally you do it if there's an ID, you go on the IDs. But because we had a lack of data for in in one of the tables, in the job title table, we decided to use a different column to to join on, and now we're able to look at all the data together.

So super quickly, that is an inner join, a left join, a right join, and a full outer join, and it's pretty easily visualized here, and you're able to uh change what you're joining on right here, but you're also you can do multiple. So if we want to do the employee ID and the employee ID, you can do that as well, and you can keep going as as many as you'd like um and right here, or you can change some of these things. Uh I don't there aren't a lot lot of use cases for this um but you know you can absolutely do this um and mess around with this, as seen. I'm not going to go through it in the tutorial because again, 95 plus percent of the joins you're doing, you're going to want to do it to where this equals this um and if you want to get into where it doesn't equal or all these other things, which is more complicated, I think it's much better to learn that in SQL; that's my personal preference. And so um again, all in the SQL tutorial if you want to check that one out. So you're able to join on multiple things.

Now let's get rid of that one because we can actually bring in this salary one as well, and what you'll see right down here is that we have our employee ID, and this is all coming from the demographics: so employee ID, name of employer, employee age, employee gender. Then right over here we have the job title table: so employee ID, job title, employee name, job title. And then right over here was or is our salary table, and so we have employee ID, salary and employee salary. So again, this is a way that you can put all of this data into one place, and in just a second we'll go into the worksheet right down here. I'm going to show you kind of how it looks because it looks a little bit different um than previous tutorials, and so I want to show you how that actually all works together um but again, you can create these joins um as well and do the exact same thing that we just looked at, and C, customize the joins, customize what you're what you're um uh joining on, and then you have your finished product.

And so right now we have our demographics plus Tableau joins file, and we can rename that if we want. I'm going to call this um demographics plus joins demo and click enter. And so now that is saved. So now let's go down to the go to worksheet; we're going to click on that. And so up here on our left side, this may look a a little bit different than it normally does um because it's broken out um on the measure names and the measure values; it's broken out by the tables that they were joined on. So we can pull in the employee gender now, and we can pull in the employee name now um and we can pull in the employee ID again if we want to from the job title table, and we can pulled in the employee ID from the salary table; we could do that if we wanted to. It makes no sense uh uh for actually creating any visualizations, but you know you can do that. And so you probably you wouldn't be able to do that if you hadn't joined these together. And so down here in the measure values, the values that we have from the demographics table and the salary table, all of the um all of the stuff from the employee title, none of those things were um values, and so we can't use; there are going to be no values down here. And so really quick, let's take the name of the employee, let's take their salary, sure why not um let's order that, let's take the employee salary, we'll do color and uh let's expand this out a little bit, maybe one more time, oops, just like that, and there you go.

So that is how you do joins in Tableau, and I think Tableau does a really fantastic job of making it pretty simple. They have the different types of joins when you click on that that join button, and it shows you the inner and the left and the right in the full outer, and they make it pretty simple um and and it's just really useful to be able to see that while you're creating it and see the output below like we just did a second ago; it it just makes it so simple to create those joins and then just keep going because you already know what your output is going to be, and you can kind of mess around with it and make sure you're getting the data that you need. In the very next video we're going to be doing an entire project in Tableau; we're going to be using a lot more data, and it's going to be a a complete project that you can add to your portfolio, and it's going to be a really good time. So I hope that you joined me for that one. I appreciate your time; I hope that this was helpful. Thank you guys so much for watching; I really appreciate it. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

What's going on everybody? Welcome back to the Tableau tutorial series. This is our very last video in the series, and today we'll be doing an entire [Music] project. Now, if you're watching this video, I hope that you watch the other four videos in this series, just so you can get the basics down; you kind of know what you're doing. Uh this won't be a crazy hard project; this is a beginner tutorial series, so I'm trying to make this super easy so you can follow along; nothing super complicated, I promise. And if you were wanting to go above and beyond and just make a lot of different dashboards or try a lot of different things, there's a ton of data in here, and so I'll show you some of the things that I would do, you know, as we go through it, of the things that I would be looking at and some of the different visualizations that I might do as well. But again, in this video we're going to be sticking to a lot of the basics, but I'll switch over my screen in just a second; I will show you the final product, and then we will actually walk through step-by-step of how to do the entire dashboard, and at the end you should have a completed project that you can add to your portfolio or you know just share on LinkedIn if you want to do that as well. With that being said, let's jump over my screen and let's get started.

All right, so let's get me off screen and show you what we're going to be working on today. This is the final dashboard that we're actually going to be building, and so it it's nothing crazy, right? I'm sure you have seen all these things before um and I'm just going to help you kind of build it out, show you what to do, the buttons to click um and it's really going to be a simple walk-through. By the end of this, you should be able to do all these things very easily, and I highly encourage looking at the data and looking at these visualizations and seeing what else you can do with it. There's a lot of different colors, a lot of different visualizations um that you can do with this data; I'm just showing you this today. And so the more you go out there and the more you do this on your own and you mess around with stuff and and choose different things and see how it all works, the better you're going to get, and so I highly highly encourage doing that. Uh so what we are going to be working with today is an Airbnb data set; I'm going to show you that in just a second, and I'm going to show you the data and we're going to just jump right into it.

All right, so this is the data set that we're going to be using; this is the Seattle Airbnb open data set, and let's go down really quick um there's three different CSVs in here, and so this is some of the data that we're going to be working with um some date on listings and some pricing, and then there's the actual listing that shows um the actual street address, the location, the price, the bedrooms, all of these good stuff, and then there's a reviews um and it has you know some comments and you know talks about some of the reviews. So this is what we're going to be working with, but you don't have to go in here and download it; I have already combined all of these CSVs into one; I've put it on the GitHub, so I'll have a link below so you can just click on that and you don't have to do all the stuff that I did to get this set up. Um just so you know, this is from 2016, so this data set is a little bit old. If you want to, you can come right here, and I will leave this link as well, and you can get the data set from, you know, what is this, a couple weeks ago; uh this is they they are continuing to update this; this is always updated, and so you can go ahead and download these, but some of these are these CSV Dogz um so you may need to like convert it; I don't want to go through that process um on you know in the video, and so I am just going to go with what is literally in Kaggle um and use that, but if you want to have an updated one for your project, I just advise you to go in here and grab yourself, and that's should be perfectly good. So go ahead and download the data set from the GitHub, and we should be good to go.

So this is the Excel that I was just talking about; this has all of our CSVs in one place; this is you know an Excel workbook. So in this reviews, actually let's start with the listings because that's kind of where it all stems from. Uh we have our listing, and the DAT or the data in here is um you know really extensive; there's a lot of data in here. So let's get over really quick um the listing refers to the actual home that they're renting out, the Airbnb, so it shows their location um and there's a lot more location information over here; I'm getting into it in in just a second. So there's the neighborhood, the city, state um zip code, all stuff that you know may be useful; there's a latitude and longitude; it shows what type of property it is, so that's really good um right over here; it has you know how many bathrooms, bedrooms and beds um you know sometimes if it's a five-bedroom house it has seven beds, so that's why there's those two different um fields. I don't know if you're familiar with Airbnb and and you know what they have on there, but just something to note; they have the price; this is the price per day; there's a weekly price, a monthly price, and if there's a deposit needed uh and then a cleaning fee as well; so a bunch of financial data that's you know super useful; we go into it a little bit, but there's so much you can do with that um you know if you want to dig into that, and that's kind of it; the rest of it's pretty uh pretty useless um and there's a lot; so there's so much data in here, almost you know more than half by far is nothing you would put in any type of visualization um and this is pretty common; you're not going to get data every column where you're going to be able to use it; a lot of times it's just a lot of useless junk, and so you have to know what you're looking for and know uh you know what's actually useful. So that's the listing; then we have reviews. Now what's really a little bit confusing in here and something that you just need to kind of understand about the data um and something that if you're if you get a data analyst job you need to understand your data because it's very easy to come in here and say okay there's an ID ID field and here's an ID field, so that means that those are the same; well, not in this case; um this ID field is actually the review reviews ID, not the reviewer ID that refers to like the person; this is the reviews ID; this listing ID is the actual ID right there; so really important to note um and then the L. And so then they just have their comment there, what they left as a review. And then on the calendar um I don't know why I'm scrolling down uh we have this listing ID again; so again that listing ID is equal to the ID in this listing table, and we have a date and a price, so this refers to a specific location and on this day they got $85 for it; somebody rented it out um and so then there's these like T's and F's; let's try to find a blank one; really quick, here's a blank one; so there's these T's and F's; the T means that it was taken; the F means that it's vacant; I don't know exactly what it means uh what the TNF means, but that we can deduce that much from this, and so you can see when and how much this person was making or this homeade uh in that time; so really really good data in here; there's a lot to work with um and and so we're just going to be kind of I'll give you a little bit of a use case for it in a second, and then we're going to start trying to answer some of those the building out some of the visualizations for that use case. Uh again, you could have 20 different use cases for this data or more um honestly for this data where you could build out different dashboards and different reports literally with just this data, but you know we're doing a pretty general broad project, and so it's hard to answer all of them. So let's jump over to Tableau; we're going to get started on this and we are going to build out everything.

All right, so let's come right here; this is a Microsoft Excel; we'll open that up; do this one; we will open it; give it just a second; says it's executing the query; it's pulling the data in. All right, so we have our c our listing and our reviews; those are the different tabs at the bottom; we're going to start with the listing; this is the the kind of the main one; has um you know the there's I didn't show you but there's about 3,600 locations that they had in there; let's just have it update automatically; I don't know why we need to click on that, but um so we have this listings; we have our calendar and our reviews; what we're going to do is going to come in here and we're going to open it as we did in our very last video uh for the joins; so now that we've opened it we can kind of go in here and we can do the joins as um as needed; and so let's go over here and we're going to uh let's start with calendar; going put it right there; that was super slow; I apologize. All right, let's wait for it to get the data; start setting everything up; did not think it would take this long; I apologize; no, take your time. So let's click on here, and right now it has the uh the join based on the price, which obviously is not going to work um and if you remember there is no ID in this calendar; it's just the listing ID um we can actually look right here; there's just the listing ID; so we're actually going to but listing ID is equal to ID, and right down here we can see that we have a lot of of well you can't see it um but we show that there is a lot of data um and so we know that that is correct; we know that that is now pulling in data correctly because it's showing up down here; so that's a good thing. Now in this listings there are about 3600 um about 3600 listings, and so that all the data that's in listings is going to be in there, but on the calendar, because we converted from a CSV to an Excel workbook, it isn't able to store as much information, so some of the ones in calendar may have gotten cut off, so we can just keep it this inner join because we know that if it's in listings it is going to be in calendar; we know that it if it um there may be some in calendar that aren't in listings; so if we really um you know if we really really wanted to we could do a full outer or something like that; I I haven't really thought through this as I'm talking through it in my head, but we know that uh everything that's in listing is going to be in calendar uh and so you know we don't really need to do anything other than an inner join, and we can also pull in these reviews, and it's going to do the same thing as before; we're just kind of pulling in the data, and it defaults to ID equals ID. Now we know that that is not not correct um because the ID in here is referring to the review ID; we need to go to the listings ID; so we need the ID be able to you know be part of that listings ID; if we do the ID it goes down to 2,555 rows; if we do how it's supposed and because that's just you know it's random luck there happen to be some numbers that are in both fields um that tie together; if we do the correct one where we hit the listing ID it bumps it up to I think 2,373,000, oh maybe more than that uh 23 million rows, right? A lot lot lot more, and so it's super important to get these joins right to tie them together on the right fields; if you just do it based off what Tableau tells you because it has that automated um you know it goes into these fields and says okay these are the same exact column name so they're most likely going to be what you're looking for; well it was incorrect in this point; so it's really important to check those things and make sure you're pulling in the right data; again we're going to keep it that inner join um you know if you wanted to you know try to see if there's any other data that correlate; we're keeping it simple today, but sometimes you need to join on multiple things uh so just

Uh, uh, you know a tip, so let's get out of here. Um, and we are good to go. So this is our listings plus Tableau full project; that's what we'll that's what we'll be working with. Um, and we we were able to tie all three of these, um, you know, as you call them tables or sheets or whatever you want to call them, we were able to tie them together. So let's go over here to our first worksheet. Uh, let's see. All right, so this says Tableau public only works with less than 15 million rows of data. We have 23 million rows of data; that is uh, that's a problem. Um, and when I did this before, it didn't do that, so I, you know, we're going to work through this together. So this is date reviews. I believe this is date for um um this is date for the calendar, which is going to be a lot of row of data, and so I'm sure that's part of it. Let's see. Let's do years; we only want 2016. Oops, we only want [Music] 2016. Let's do okay. Let's see what that does. Let's see if that gets us under what we need. Um, we only want 2016 data anyways, so if it's in 2017, we were going to take it out. Um, anyway, so we'll see if that gets us underneath. I have absolutely if this T ends up taking like 20 minutes, I will just cut it and, you know, you won't have to wait as long as I'm waiting. So let's see how long it takes. All right, so it took about 20 minutes and it did absolutely nothing.

Um, one thing I do know is that we don't actually use this review tables at all, um, just for demonstration purposes, so we're going to remove that and let's see if that helps us in any way. If it does, we're just going to keep it as is. Um, you know, the reviews table is really just for demonstrating how to do the joins, uh, but we weren't actually using any of the data for any of the visualizations, although you could. Again, I want to see how long this takes, uh, and I'll cut ahead. All right, so that worked uh perfectly; it apparently took out all the data that we needed, all the rows that we needed to get under that level. Again, I was just doing that to show you the the that joins, how you needed to change the columns to make sure that it joined properly. We don't actually use it for any of the visualization, so their end product is going to be totally fine. I don't know why uh this didn't happen to me when I when I created this whole thing already. Um, so just going to move forward because I make mistakes, so uh let's keep moving. The first one that we are going to make is that uh is that colorful one. I'll probably pop it up on screen so you can see it. Uh, well, if I remember, I'm going to pop it up on screen. Um, it's the colorful one; it's the price by ZIP code, so we're going to be looking at these zip codes and kind of see um, you know, how expensive is each zip code.

Um, and before we actually start, I just remembered I want to talk to you about the use case for this data. I want to imagine you to imagine that you're working for somebody; they're like, hey, where, you know, I want to start an Airbnb business. I want to know where I should go, where should I buy up buy a home, put it up on Airbnb and start renting it out? Where's the best place? You know, what are some of the factors that I should be looking at? Uh, and so that's kind of what our use case is. So we're going to some of the things that he cares about are things like bedrooms, um, location, which is really important, and how much price he's actually going to get; how much money can he charge? And so he's trying to optimize that to make sure that whatever rental he gets, he can make a the most profit from instead of choosing something that, you know, he thinks would work, but, you know, in the end, he's actually not making that much money. So those things are important. So that's our use case; we're trying to help this guy out, help him find a really good Airbnb. Um, so let's take a look at these zip codes real quick. We have uh quite a few of them, and there's one that's null. Uh, we'll exclude that, or if if it doesn't have a zip code, we'll just exclude those because they're not going to show up on the these visualizations anyways. Um, and so we want to look at the price, so we just want to find uh the price, which should actually be down here and not the sum. Uh, no, we want to look at the average price, and let's order that. This is great. Um, so this is the most expensive one, ZIP code 98134, at $26 uh per for the average price. Uh, but let's give that some color really quick. Let's uh where's the ZIP code? It's up here, so let's take that zip code; we're going to put it right over here. We going to do color, and it's going to give it some uh assorted colors. Now these colors are going to um, when we do map in just a little bit, these colors will um match what we're doing in there, and so, you know, I I like to try to color coordinate things. Um, we're not doing going too crazy with the colors today, so this is our very first visualization. Congratulations; it is uh it is complete. So uh we can label this one, and we can just do price by ZIP code, and I'll make that bold. I don't know; I usually like a bold. We'll apply; we'll do like that, and boom, first one is done. Uh, and this is our starting place to say uh, Hey person who's looking to buy this Airbnb, here are the zip codes where they are able to charge the most um for for their Airbnb.

So let's go over to the second sheet, and we are going to be doing the map, and so um map is pretty easy, but it it's pretty easy. Once you actually get the data that you need, all Al though there's a lot of different data that you can use for the actual U map right here, you need something that shows um the location, and there's a lot of things that show location in here. In fact, they already um provide a latitude and longitude, and then at the bottom, they generated a latitude and longitude from from some different um fields, and then there's just a bunch of different um State; there's um States; there's zip codes; there are uh I think another one, yeah, like country; there's a lot of location data in here, so which one do we want to use? We want to stay consistent; we don't want to deviate from that and start using different um L long longitude and latitudinal uh coordinates because that could throw off our our results completely. We want to stay consistent with what we're using, so we actually want to use this ZIP code, but when we pull it up here, it's going to give us uh basically the same um, you know, it's going to show these zip codes, but we were going to right over here; we're going to click on this one, and now it's going to separate them out. So now we have all of these um, you know, kind of separated out. What you might get when you first do this um is it might look like this; you may have to zoom in. Um, I know that that happened to me the other time. EXC me, go to here; that's what happened to me uh just when I first did it. So uh know that that may happen, and we want to change the colors the exact same way that we did them before, so we're just going over here; we're doing color, and these colors do um they do should match up with the um with the other ones. Let me um exclude this; let me see if it does. 98134, that's the blue, and right over here 98134, that's a blue. I I I believe they are going to be the same. Yep, and so just scrolling back, if you look at the zip code on the far right, uh, they are the same. So if you look at like this section right over here, I I'm just wanting to make sure I'm not going crazy uh before I get into this and realize I'm not correct at all. So uh now what we want is, you know, this doesn't really give us any information. If I was just to glance at this map, I would have no idea what you're trying to show me um any information off this, so we want to show some actual information. So first thing that we're going to do is we're going to actually add the label to this so that you can see it, you know, when you're going over here and you see okay, here's this um zip code um in the dashboard when we create it, you can click on this, but if you just want to do it visually without having to click anywhere, you'll be able to see okay, 98134, that's right here, so this location right here is, you know, able to charge a lot of money; it's probably a really nice neighborhood. So U and we can back that up by putting the average price. So these these two visualizations are really they really go hand in hand. We're going to add oops, not the sum; this one needs to be the average, so you go to this measure, the sum, go to average, and there you go, and these should match, so this should be 206.000. Um, I'm looking at the average price right here, and then we go over here 98134, 26.6, so this all matches. Um, and we can uh we can actually change that size a little bit bit if you want to actually get it in um get it within each of these things, you know, adjust it as you see fits. I think that's fine right there. Um, no need to mess with it anymore. All right, so let me see; I think that is everything for this one. I don't know if I want to add anything else. Uh, no, I'm going to keep it how it is, so that is our second visualization. Again, these ones are directly uh correlated and and, you know, this there's just different ways to visualize it. This one you can see actually on the map where it is and the average price; this one you can see from highest to lowest. So again, you know, sometimes when you're doing these visualizations, you're going to have these accompanying um uh these accompanying visualizations in your dashboard; that's very normal.

So let's move over to the third one, and for this third one, um, you know, something that our guy was looking at is he's like, okay, well, you know, I'm thinking about listing it on Airbnb, but I also want to live in it, so I want to know the best times to actually um, you know, put it on the market for people to be able to use, and so I was like, okay, man, no problem. Uh, let's let's take a look at when when are people spending the most money in airbnbs, and we actually had that calendar. Um, if you remember, let's look let's see this calendar, so we have this available, the date, the listing, all of that stuff. Um, and let's look at the date in here. Uh, and we obviously don't want it like this; we want it to be more uh more of a Time series, and we're going to do be doing that based off of uh the price for the calendar. So let's go see if we can find that really quick. Okay, here's the price; where is that calendar one? Let me see. Okay, there's the calendar. Oh, here; I totally forgot where that was supposed to be. O, that looks terrible. Okay. Um, let's see. Let's let's start working on this because this needs some work obviously. Uh, this is the worst visualization I have ever seen. Um, so we need to work on this a little bit. What we need to do is we need to change up whoops; we need to change some the way that these dates are are seen. So right here is a these are two separate things, so if I go right here and I Do by quarter, it's just going to change the quarters here, right? That's that isn't really helpful. We actually want to keep the year here. What we want to do it is by year; we want to separate it by year, um, but we want to separate it. Let's just do I don't know; let's try weak and see what it looks like. Okay, this is great; this is this is what we're looking at again. Um, if we went back and Chang this like quarter, it change to quarter and then change it to week, it would show the quarters, but it wouldn't show everything, right? This isn't all the data that we need, and so, you know, you really need to make sure that you're doing this correct. By default, it's almost always year, but if you're looking at it via quarter, so like let's say somebody comes in, you say, hey, what quarters I want to I'm break these out by quarters um and not year-over year; that's how you would do this, but in the year, we want to break it out by uh the week, and you see this huge drop off um at the end. Well, that is actually because the data doesn't go past that; there's just like one day of data or one one um week of data in here with actual um with January of 2017 data, so it just drops off because this is an this is a sum, so it only adds up to like um 591,000 compared to like the 2 million, so we want to get rid of that. Um, and how do we do that? Uh, let's see. I think it's filter; is it format? No, it's not format. What am I thinking? Bear with me. Uh, let's a filter. Well, I was looking for it; I just couldn't find it. Uh, let's bring it back to the 31st. Let's see if that fixes what we need. Perfect. Uh, that's all you had to do. Um, and the reason that this is helpful, and often times you'd have several years worth of data in here, um, and then you could have you could do even do something like this, um like this one where it has multiple lines, the reason that this is helpful is because if I'm telling my friend, let's I mean, just I'm going to say it's a friend or business partner, whatever you whatever you want to use this use case for, I'm going to tell him, hey, the beginning of January all the way until like, you know, even February, it's like really low; it's half, so there's not a lot of people traveling because everyone travels when at the end of the year, so in November, December for the holidays to visit family, um, and then in the summer for vacations. I would tell him just based off this one thing; I would say, hey, over the summer and then at the end of the year and during the holidays, that's when I would be renting out your Airbnb. Okay, so just this one very simple visualization can help him understand the best times um to do that. That may been intuitive; you may have already known that, but you can prove it with the data, which is always really helpful. Um, and let's see; is there anything else that we need to do with this? Uh, I'm just going to label it, and I'm going to say um revenue for year. Let's do bold; do apply; there we go. Do I label this last one? I didn't. Let's label that last [Music] one, and we'll do price per zip code; price per zip code; we'll just keep it at that; keep it simple. Um, and let's do that. All right, I believe we have two more, so we have done um we've done three of them. Um, we got the zip codes; we've got the um, you know, the time of the year. Now something else that he was wanting to know is um, you know, just how things affected, and something that's going to affect the price of the actual Airbnb is going to be the amount of bedroom, so the the larger the house, the more bedrooms, the more it's going to cost typically, so we can take a look at that. Let's pull in these bedrooms, um, and that will be our columns. Uh, no, it won't. What we need to do um and so I I knew this was going to happen; I just forgot it until right uh until right now. What we this right now is actually a um it's a a value, right? So it's a number, and that's totally um reasonable because if we go right here, we do count distinct; that's because there's only seven values, right? It goes; there's zero bedrooms, 1, 2, 3, 4, 5, 6, 7, all the way up to seven bedrooms. Right now, it has it as a numerical value; we want to um change that to create it as um these measure names, not a value, so we're going to um we're going to remove this; we're going to go right down here, we and click this drop down, and we're going to say convert to Dimension, and so now we're going to add it as a dimension, so there that looks um much more normal. I really quick; I'm going to I'm going to keep these in here for a second, but we're going to get rid of these nulls and zeros because if a home has zero bedrooms, that's a problem, um, and so we want to look at the price again. Let's go down here in the listings; it should be the price. Now this is the price for the location per day. Um, if you want to look at monthly or or, you know, stuff like that, they have that data, um, but we're just going to do the price, the average price, not the sum, um, although this is helpful. So just really quick before we change it, this is going to show you which which ones make the which ones are bringing in the most money; it also may show you which ones are the most common. Those are all different visualizations that we can do, but the one that brings in the most money uh that brought in 63 or that has $63 million worth of um worth of listings, so they all add up; those one bedrooms are doing phenomenal; half of that or two bedrooms at 30 million, three bedrooms at 18 million, and so on and so forth. So there's a ton of one-bedroom ones; we may even keep we could even keep that in there, um, you know, if we wanted to, um, and then we do something similar later, but you can keep something like this in there. What we will do really quick though is we're going to do the same thing that we've been doing is keeping average, um, and we are going to get rid of this because if it doesn't have the bedrooms, you know, that's not helpful to us, and if it has zero bedrooms, that's that's genuinely a problem. I will not be renting an Airbnb with my family uh that has zero bedrooms in it. So now we have this, and would be really helpful to be able to see that in the visualization. I mean, it's just kind of hard to see it as is. I mean, it just does not hurt to add that right here, do a label. Um, why is it angled like that? Maybe I just need to move it out more. That looks much better. Um, that's the average price; that cannot be right; that's the sum; that's why. So let's go over here; let's make that average as well. Much better because uh if the price was $3 million for a three-bedroom, I would not be going there. So this is really really useful information for our friend, right? If um he wants to, you know, get into those one that one bedroom area, you know, you're not going to be making a lot of money; it may be low cost UPF front, but he's not going to be making a lot of money. It significantly goes up when you reach these five and six bedroom homes, which makes sense. I mean, if it has five or six bedrooms in it, it's probably a really large, really nice home, and you can charge a lot more money, and our friend is uh extremely wealthy; he can buy whatever he wants, and so he may be looking at these um larger ones, seeing that there's a much higher return um on his investment the higher and the more bedrooms he goes. So we're going to keep it just as it is. Um, and let me see; is there's anything else that we want to do with this? No, we're going to keep it just like this. Uh, and the last one is by far the easiest, and we actually just discussed it a little bit. We want to know, you know, what's his competition look like, so um for those for the bedrooms specifically, so let's

Go back up to the bedrooms. We want that one to be right here in our rows, so we show um these, and then we just want a count of um how many listings there are. So we can do that via the listings ID. So here's our listings; each ID represents one location or one home. So we're going to do that right here. Uh, that looks absolutely terrible. That looks terrible. What am I doing wrong here? Um, let me see. Uh, one thing we need to do is we want to get rid of these nulls and zeros. Do that really quick. Um, and then we don't want to do just the ID because I, I'm realizing now uh what I'm doing. I need to convert this to a numeric so we can do a count on it. So let's um oops, let me see what, what is happening? This is terrible. All right, let's put this back. Let's make, let me see if I can just um do an attribute. Let's do, do the [Music] count and let's do text. Um, no, it needs to be a distinct count because that's, that's basically like um a count of the numbers themselves, not each individual ID.

Okay, it took some figuring out. I'm going to keep that in there because you guys need to see uh a lot of you guys like seeing when I make mistakes, so you know, makes it feel like when you make mistakes it's okay. Um, and I'm all about that, so I'm leaving that in there. You guys can see me fail a little bit. Um, I just forgot how to do that for a second. And this is exactly what we're looking for, right? We want, we now it showed us in that visualization that we were looking at earlier before we um switched it to the average price. This is showing us that there are for one bedrooms, there's 1,800; one bedroom two, that 483; three, that have 206; four, that 55; only five that have 20; and six that have five. So the more you go up, the less and less it is, or the less and less competition there's going to be. Now, is there a lot of demand for four-bedroom, five-bedroom, six-bedroom? Uh, that's for our friend to figure out. Um, well, maybe we'll help them out with that later um in the with the data. You know, we could look at the reviews that we had. Um, there's so much data in here, and we could absolutely figure that out. But for what it's worth, we're giving him this initial stuff, and he'll have follow-up questions for us later. That's how it always works. I promise.

So now we're good with this one. Let's label this one. Did I label the last one? I will go back and look. Um, distinct, I, I'm going to butcher this one. I'm do distinct count of, of bedroom listings. I don't, that may not make sense at all, but we're keeping it. So we're going to do bedroom apply. Okay, let me see if I added the label on this one. I didn't. Let me do that real quick. We'll do average price per bedroom. Again, I'm oops, you didn't see that. I'm just going with whatever is coming to my head. This probably wouldn't be what I would keep if I this were like an actual project, but it works for now. So we have our five visualizations: 1, 2, 3, 4, and 5. And let's create our dashboard. That's going to be this button right here. So we're going to click that. We are going to uh go right here, and we're going to say automatic, cuz we want to use this entire area. And so now we're just going to start um, you know, pulling them over, and I'm just going to start from the very first one and go to the very last one. Keep it really simple. So this very first one, we'll pull it over. It, you know, it's going to take up the entire space until you start adding all the other ones. We'll include this one right, right here. Um, and well, let's leave it as it is. You know, we'll adjust it um once it gets to its final place. Now we have number three. We'll add this one on this side. It looks terrible right now, but give it a second. Uh, then we have number four. We're going to add that across the top. Okay, it's already starting to look a little better, and um maybe I, you don't have to keep this in here, um but you definitely can. Uh, let's start to adjust things a little bit. Oops. Okay, let's see if I can zoom in one more. No, I'm going to do it just like that. Actually, let me [Music] see if I can make it even just a little bit closer. Perfect. Uh, that's the best you're going to get. Um, if you didn't see, I use this um magnifying, and then I could click on the area that I wanted to see. So we're going to keep that just like that. We're going to move this over because that is um definitely not as important. Um, and then we're going to move this way over as well. So keep it just like that. Again, this is something where if you want to, you can click on this. Um, it didn't, I don't know why uh I can't remember how to get those connected, but it's you definitely can. Um, but oh, okay, I was just clicking on the wrong one. That's why, that is why. But you can click over here, and you, you know, it'll filter um based on, so if I go to this one, oops [Music] dang, oh jeez, what am I doing? Oh, this is a travesty. Okay, let's try to get this back. All right, I'm not touching it, guys. You get the gist. You can mess around with it yourself. I'm not messing this up. Okay, so the next thing we need to add is the very last one. It's going to go right up here, and then we're just gonna kind of move it off to the [Music] side, and let's see, going add, yeah, have this caption. Um, if you've never seen something like this before, um and I actually want to make this bigger as well. Jeez, give me a second. It's, it's kind of lagging a little bit, and make this a little bit tall. Maybe I don't want it as wide, but I definitely want a little [Music] taller. Give it a second. Yeah, let me scooch this [Music] back just like that. That's fine. Uh, we can keep it like that. In my original one, I didn't have this. Um, you can get rid of this if you want, you know, you can um um, you know, just exit out right here if you want to do that. But there you have it. Uh, this is the entire thing. So we started from the very start. Um, we started with this one, then this one, uh did some, um and this is, you know, all the zip, all of our ZIP code work. Then we took a look at the calendar where we looked at the price and did some time series visualization, and then we're looking at the bedrooms and and the count of bedrooms. And so this should be really helpful for our friend. This should be an initial dashboard to get him going. And once he sees this, he's going to have a million other questions, and he's going to want another dashboard for different data that's in there. He's going to ask about, okay, well, what if I want to do it weekly, or you know, I want to rent it out for the month, or you know, how many um reviews are people, five-star reviews are people giving on, you know, one bedroom, two bedroom, three-bedroom. These are all things that, you know, he may ask, and then we'd have to build out. In the real world, this is what happens all the time. You know, they make a request, and then they're like, oh, this is great, but I also want this. So um, you know, your friend is, is going to be right in line with just about everyone else um that has ever gotten a dashboard uh for work or for personal use.

With that being said, this is it. Um, we have done the entire thing. Now, if you want to share this, it is super, super easy to share. Um, and I'm going to try to remember how to share it. Uh, so we're going to do save to table public As, and we're going to do this, and we're, we're going to make it um let's do Air BnB, is it like, is it a capital B? Is it like that? No, that doesn't look right. Airbnb. Uh, we'll do full project, and we'll save. And that is being created right now. Um, and I will save this, so if you guys want to go look at this, you can. Um, and I'll provide a link in the description as well for that and see if yours looks um similar to mine or better than mine. Give it a second because it's thinking. All right, so here it is. So here's our final, our final project. Um, and if you followed step by step, then you should get this exact or very, very similar to this one. Again, I encourage you to, if you want to have the up-to-date data, to go to that um Link in the description that has um the the most recent data, and they update that I believe monthly. So you can go there, get the most recent data, and then you can do stuff, and you can create a beautiful project just like this, um but with the, you know, the most recent data. Again, I use the Kaggle data just so you guys can remember, and I encourage you to look at the different data points that are in the Excel. There is so much in there, and you can use uh honestly, like there's probably 30 or 40 other fields that you could be using in there that we never even touched. Um, but for this project, we were keeping it pretty simple. And so go do that, make completely unique dashboards and and visualizations and create projects and add it to your portfolios so that you can create a fantastic portfolio website and get a job. And that's what this is all about. Um, it's about upskilling and and getting these skills that you can, you know, get a job or or do better in your job. So I hope this has been helpful. I really appreciate you guys joining me and and doing this entire project with me. I have no idea how long this is. This probably, this could be like an hour for all I know. Um, so thank you so much for sticking with me this entire time. If you like this video, be sure to like and subscribe below, and I will see you in the next [Music] video.

What's going on, everybody? Welcome back to another video. Today we're going to be starting our Power BI tutorial [Music] series. Now, I am super excited to start this series with you guys. Guys, we are going to be breaking this up in about six or seven videos. I don't really like those super long videos where it's like 4 hours long. I like breaking mine up into chunks, so that's what we're going to do. This is the beginner series, and so we're going to start with the very Basics, and we're just going to work our way up, and I'm going to walk you through every single step of the way. It'll be very easy to follow. Everything will be provided for you so that all you have to do is really follow along. And by the end of it, you should know Power BI a lot better. You should have a lot more confidence using it. Now, before we actually jump onto my screen, I want to give a huge shout out to these sponsors of this video, and that is Udemy. You guys know that I absolutely love Udemy. I've been using them for years, and that is no exception when it comes to Power BI. I have taken some of the best Power BI courses ever on Udemy, so I highly recommend you checking out the ones I have in the description. These are ones that I actually took, and I loved the most. So if you're looking for a full Power BI course, I highly recommend checking out Udemy. Thank you so much again to our sponsor, and now without further ado, let's jump on to my screen and get started with the tutorial.

All right, so the first thing I'm going to do is download Power BI Desktop. I will leave this link in the description so you can just click on it, go to it, and download it. We're going to click this download free button, and once we click it, you can go to the Microsoft store, and I already have it downloaded, so when you see it, uh it'll already say downloaded, but um for you, you can go in here, you can click download, and it will download it for you. I'm on Microsoft, uh but it may look a little bit different for you if you're on a different system. But once that is done, we are going to open up Power BI. So let's go right down here to our search, let's go to Power BI, and it is going to open up for us. All right, so right away, this is what it's going to look like when you open it, and we're going to go right over here to get data, and let's click on that. It's going to open up this window, and it's going to give us a lot of different options for where we can get data from. Now, some of these are free, and some you need to upgrade from, but you just taking a quick glance through here, you have a ton of options. There's databases, there's um, you know, blob storages, there's PostgreSQL or different SQL databases, um there's Google Analytics, there's a lot of places, and you can go through the process to connect to that data, and you can pull that data in from those data sources. Now, for what we are doing, we're just going to be using an Excel. I'm going to leave the Excel that I'm going to be using in the description. You can go and download it and walk through this with me. So what we're going to do is click on Excel workbook, and we're going to click connect. So we're going to go right here in our Power BI tutorials folder, and we're going to click on apocalypse food prep. So let's click on that, and it is going to connect and pull that data in. Now, right here we have our Navigator, and so if you had a lot of different sheets, you can click on that and choose which ones to pull in. I just clicked on it right over here, and we're able to preview the data, but I can't load or transform it yet. I need to select which sheets I'm bringing in. So we only have one; that's the only one we're going to bring in. So you can go ahead and load the data, or you can click on transform data. It's going to take us to Power BI Power Query, which is going to allow us to transform our data. So I'm going to have an entire video on how to transform the data, but I'm going to give you a really quick glance at it to kind of show you what it is. So right up here it says our Power Query editor, and this is a the window to basically transform your data and get it ready for your visualizations. Now, you can do this in Excel if you want to and do that beforehand, or you can do it here, and there are lots of things that we can do in here, as you can see at the top. Again, I'll have an entire video dedicated to just Power Query, but let's take a quick look at the data and see if there's anything we want to transform quickly before we actually go and start building our visualizations. So over here we have the store where we purchased it, we have the product that we purchased, the price that we paid, and the date that we bought it. Now, the first thing that jumps out to me is that this just says date on it. Um, we might want to say date_purchased, and we're going to hit enter. And if you noticed right over here on these applied steps, it says renamed columns. Everything that you do, every single step that you apply to transform this data is going to be right over here. And if I want to, if I go back and I say, you know, I really didn't want to rename that column, I can just click X, and it is going to get rid of that and take it back to its original state. So again, I'm just going to say purchased, and we're going to enter that. Now, this is our apocalypse food prep, so this is food that we are buying for the apocalypse um for this example. And if we look at our products, we have bottled water, canned vegetables, dried beans, milk, and rice, and all of that stuff makes sense except for the milk. Uh, milk will not stay or last long in the apocalypse, so I think what we're going to do is we're going to filter that out really quickly, and we're going to click okay. And right over here, again, says filtered rows. And so now if we scroll down, there's no milk. So what we're going to do is we're going to go over here to close and apply, and it is going to actually load the data into Power BI Desktop. So on this left-hand side, it immediately takes us to the report tab, and what we want to do is go right here to the data tab and take a look at our data. So again, there's our date purchased, and as you can see, the milk is not in there. Another tab that we're going to take a look at, um and again in this report tab, this is where we actually build our visualizations. The data is where we can see the data and and change it up a little bit and change some small things about it, like sorting the columns or even creating a new column. And over here we have this other tab, and it is called model, and this is especially useful when you have multiple tables or multiple excels, and you need to join them to kind of connect them together. We don't have that, but in a future video, I'm going to walk through how to use this entire tab. So now let's go back to the data tab, and I want to just look at the really quickly before we go over to the report tab and we start building our first visualization. As you can see, I've been buying these different products in different months, so this rice I've been purchasing in January, February, March, and April, and I've been buying it from three different locations because I wanted to see if I was spending less money at one location on all of the products, then I would just shop there in the future and save a lot of money, or if there were specific products that were really cheap at one location, but others they were cheaper at a different location, so I should just buy like the dried beans at Costco, but everything else I should be buying at Walmart. And so that's what we're going to look at in just a little bit. So let's go over to the report tab. Right up here at the top, there's this data section, so you can kind of choose if you want to add any more data. Now that we are here, we can also write queries or transform the data like we were looking at in the Power Query editor window. Over here in the insert, we can add a new visualization or a text box, and then in the calculation section, we can create a new measure or a quick measure, and then over here we have share where you can actually publish your report or your dashboard online. Now, over on the visualization section on this far right, this is a very important area; this is where a lot of the actual creating of the dashboards happen. So let's take a look really quick, and we'll get into a lot of these things as we're actually building our dashboard, so we're not just sitting here looking and talking; we're going to be actually building and doing. All right, so we're going to click right here on this drop down on sheet one. It's going to show us all of our columns. Now, two of the things that we wanted to look at were where are we spending the least amount of money buying the exact same product? That'll help us determine where we want to shop. And the second thing was, should I be buying all my products at the same place, or are there certain products that they're going to be cheaper at a specific store, and I should buy it there? So let's start out with the first one, which we're just going to see uh with the store and the price uh where we're spending the least amount of money. And just at a quick glance, we can see we're spending the least amount of money at Costco at $210 versus Target 219 and Walmart at 225. And that really answers our question, but we want to visualize it better, be able to see it in an easier way. So we're going to go right over

What's going on everybody? Today we're continuing our Power BI tutorial series, and in this video, we're going to be looking at Power Query. Now, Power Query is really great because it allows you to actually transform the data before you actually get it into Power BI. So, if you want to make any changes, like adding or deleting a column, or changing the data type, or a ton of other things, you can do all of that in Power Query. Now, without further ado, let's jump onto my screen and get started with the tutorial.

All right, so before we jump over to Power BI and start using Power Query, I wanted to take a look at the data, and this is the Excel from our last video called "Apocalypse Food Prep." In that video, we went through and we bought some rice, some beans, water, vegetables, and milk, all for the apocalypse—getting prepared for that. Now, we decided to buy some additional things like rope, some flashlights, duct tape, and a water filter—several water filters. And after we purchased those, our boss, or whoever we're working with, or somebody, decided to go and make a pivot table. Now, in this pivot table, they kind of broke it out by Costco, Target, and Walmart, and had all the items, had some subtotals, as well as some grand totals right here. And then they decided to kind of copy and paste that into this, and you'll see this a lot when you're working with people who use Excel; they like to kind of make things like this, maybe make it into like a table, or format a little bit differently, but you'll see stuff like this a lot. So this is what we're going to actually pull into Power Query and work with. Now, we're going to imagine that this is all we have; this is the only thing we were working with, and I'll kind of reference this pivot table a little bit, but we're going to pretend this is all we have and we want to transform it to make it a lot more usable, to where we can make visualizations with it. So let's hop over to Power BI and pull this Excel in.

So what we're going to do is click "Import data from Excel," we're going to click "Apocalypse Food Prep," and click "Open," and then it's going to bring up this window right here. Now, this is where we can choose what data to bring in. So we can take a preview and just click on it real quick, and this is the pivot table that we were looking at, so it does have that pivot table. So we are able to pull in just a pivot table, and then we have the "Purchase Overview," where was kind of that formatted thing that we were just looking at with all the colors. We're going to pull both of those in. So we're going to pull in the pivot table and the purchase overview. Now, we could just load it, or we could transform it, and we're going to click "Transform," and that's going to bring us to Power Query. So let's click on "Transform data."

So now, really quick, before we actually jump into working through this and transforming it, I want to show you what the Power Query editor looks like. So if we go right over here, we have our queries, and these are the tables that we actually pulled in, and we can click on those and kind of go back and forth between them. Now, up top, we have our ribbon, and the ribbon offers a lot of functionality. We have things like "Remove columns," "Keep rows," "Remove rows," "Split columns"—these are these are all things that we're likely to use when using this Power Query editor. There's also another tab called "Transform," where there's a lot of functionality here as well—things like unpivoting a column, or transposing columns and rows, and using a first row as a header—some of the things that we'll be looking at today. There's also another tab called "Add a column," and this one's pretty self-explanatory, where you can add additional columns, like deleting a column, creating an index column, or a conditional column—those are the three main ones. There's also "View," "Tools," and "Help," but we're not going to really be looking at those today. And then on the far right side, we have our query settings. You can do things like change the name, so we call it "Pivot Table 2022," and it'll update right over here on our query side. And we have our applied steps. Now, our applied steps are extremely important and very, very useful. Anytime we make any change to transform this data, it's going to be documented right here, and then we can go back and look at it, or we can even delete that change in the future if we want to, and go back to a previous version of what we just did.

So when we loaded the data into Power BI, it did a few things for us: it chose the source, the navigation, and it promoted the headers, and then it also changed the data type. So if we want to check, we can actually see those things or change those things, like this "Source" right here. We can click on this little icon, and it's going to bring up the actual path where we got this file. So if we wanted to change that, or it changes in the future, we can come here, and we can change this file path, but we're not going to do that right now. So let's click on "Cancel," and let's go back down to "Change type." So I promoted these headers, and obviously these headers are not correct; we're looking at this pivot table and not the purchase overview, but it changed these column headers. And so in the future, if we wanted to, we could easily change those, but it did that for us, and it changed the type as well. So if you look right here, it says "abc123" all the way over here to where it just says "ABC." "ABC" means it's only going to be text, where "abc123" means it could be basically anything—text or it could be numeric. So now let's go over to "Purchase Overview," and this is the one that we're actually going to be working on the most, but we might be looking at "Pivot Table" just a little bit to kind of reference it and see some of the differences.

So before we do anything, let's just take a look at how Power BI decided to take this data in. So it chose this "Apocalypse Food Prep Overview" as kind of the first column, and that was kind of our header or the title of what we were looking at before, and then all these other columns are basically Column 1, 2, 3, 4, 5. So that's something that we're going to want to change in just a little bit. There's also all these blank columns right here at the top and kind of these null values as we go along, and we'll take a look at those, and we kind of are going to want to get rid of some of this and just clean this up to make it more usable for our Power BI visualizations. This may be perfectly fine and acceptable in an Excel, but when you're pulling it into Power BI, the real reason you're pulling it in is to create visualizations, not just it to look good in an Excel. So we're going to need to clean this up quite a bit. So let's go right up top. The first thing that I want to do is I want to get rid of these top rows. So we're going to go to this top ribbon, and we're going to click "Remove rows," and we're going to select "Remove top rows," and we're going to select "two" because we have one, two rows of all nulls, and those are completely useless; we just want to get rid of them right away. So let's click "OK," and it removed those. The next thing that we want to do is these—this "Location," "Product," and all these dates—these are actually the column headers that we wanted. So what we need to do now is we want to go over to "Transform," and you want to say "Use first row as headers," and just like that, we have "Location," "Products," and these dates as our headers, exactly how we wanted them. Now, let's say for whatever reason, you know, we made a mistake and we needed to go back, we would just select "Remove top rows," and that would be perfectly fine. Now you can see over here it promoted the headers, but it's also changed the data type. So before, if we went to—before we removed the headers—these were all "abc123," "abc123," because it had a lot of different data types in there, so it just kind of made a generic data type. But when we promoted these headers, the first thing that it decided to do was also change this data type for us, giving us its best guess as to what this data type is, and it decided to do this "decimal." So this "1.2" is a decimal, but we're actually going to change that, and all you have to do is click on this "1.2," or or the data type that it has right here for you, and we're going to click on "fixed decimal number," and let's do "replace current," and now it's just a little bit better. So now it's 2.70, 2.5, and that's normally how we would read values like this because this is money, so we would normally read it to the second decimal just like that. And if we have it on the second decimal for some, we should probably have it on the second decimal for all of them. So really quickly, I'm going to go through, and I'm just going to change that, and it should be pretty quick, so hang with me for just a second.

All right, that is perfect. Now, for the purposes of what we're about to do, we don't actually need these subtotals or this "Costco Total," "Target Total," and "Walmart Total," as well as the "Grand Total." Really, we want to get rid of those. And so what we're going to do is we're going to go right over here, we're going to click on this drop-down, and we're going to try to filter this data before we actually load it into Power BI. So we're going to filter, and we're going to say "Remove," "Remove empty," and let's remove those, and it's going to take out all of those nulls. If we wanted to try to filter this out by saying something like "Costco Total" or "Target Total," we could do that by going right here, clicking this drop-down on "Products," going to "Text filters," and saying "does not contain," and let's do "insert," and we're going to say "does not contain," and we want to say "Total," and let's click "OK," and again, it filtered out all of those things. So there's a few different options that you can do if you want to filter out rows that contain either null values or specific values. Now, the next thing that we're going to do is actually get rid of a column—this "Grand Total" column. And so what we're going to do is we're going to click on the very top part where it says "Grand Total," we're going to go back over here to "Home," and we're going to click on "Remove columns," and it says "insert," that's because we're on this "Filtered Rows" one right here, um, but what we're going to do is just insert that, and it'll insert it right there; that's totally fine; we can just move it to the bottom. Now we got rid of this column entirely. Now this looks really good visually; I like how this looks; I like how everything is set up. The biggest thing about this is that when you're actually wanting to use this for visualizations, these columns as dates doesn't really work too well. And so what we're going to want to do is we're going to want to transpose this or pivot this to where these dates are actually rows. So what we're going to do is select the first date, which is January 1st, all the way through April 1st, and we're hit "Shift" and click on that April 1 right there to select all of them at the same time, and then we're going to go over here to the "Transform" tab, and we're going to click "Unpivot columns," and let's see what this does. And so now what we've done is we've basically recreated our original Excel that we had. So let's go back and take a look really quickly at that. So this looks almost identical to what we have in Power BI right now, and this is extremely usable and very good for visualizations and is much, much better than this, but again, we were pretending—pretending that this is what we were given at the beginning. So you have to imagine, you know, somebody just handing you this, and you need to make it much more usable for visualizations in the future, which happens a lot, and we actually wanted to create this; we just weren't given this.

Now, a few last things that we might want to do is we want to clean this up just a little bit. We're going to select the data type and change this to "date," and then we're going to select the "Value," and I double-clicked on the "Value," and I actually want to call this "cost," or "product cost," "product cost," and then for the "Location," I actually want this to be called "Store." So now this looks really good, but I want to show you one thing really quickly on this "Pivot Table 2022." So let's go back here. This looks very similar to how we had it when it first started. One thing I wanted to show you really quickly, and I want to click on this first one, we're going to make this our column header, and then we're going to try to pivot or unpivot this "January," "February," "March," "April." So really quickly, let's do that. So we're going to "Transform," "Use first row as headers." So now we have this "January," "February," "March," "April." Now, if you notice, these are not dates; these are actually text; it says "January," "February," "March," and "April." So if we go to do this and we click "Unpivot," and here's the columns that are created when we unpivot it, it is "January," "February," "March," and "April"—these are not dates. So we cannot go and change this to a date because that would error out because it's actually text. So it's something that you want to look out for; it's something that you need to be aware of, and you can change that in the pivot table. So you want to be aware of how it actually sits and looks in the Excel or whatever data source you're pulling from before you actually pull it into Power Query to transform. And now the very last thing that we need to do to finalize all of this is go over here to "Close and Apply," and once we click that, everything that we've worked on is going to be applied to the actual data, and it's going to load into Power BI to create our visualizations. So let's go ahead and click on that. And so now the data has been pulled into Power BI. Let's go right down here to "Data," and we can see the data right here. If we need to transform this data again, we can bring it back into the Power Query editor window by just clicking the "Transform data" button, and it's going to bring us right back. So I hope that this was helpful. Thank you so much for watching. If you like this video, be sure to like and subscribe below, and check out all my other videos and everything data-anal related. I'll see you in the next video.

What's going on everybody? Welcome back to the Power BI tutorial series. Today we're going to be taking a look at building relationships. Now, when you import multiple tables from either the same data source or multiple data sources, you want to tie them together so that when you're creating your visualizations, everything is connected. So in this tutorial, we'll be walking through how to create those relationships to make sure that all of your tables are connected properly. And without further ado, let's jump onto my screen and get started with the tutorial.

All right, so before we jump over to Power BI and start creating our relationships and our model, I want to take a look at the data in Excel. We realized we were buying so many products for the apocalypse that we decided to start our own store, and we have several customers and some client information down here.

And so, I wanted to take a look at some of the columns and these tables that we're going to be looking at.

First thing we have is the apocalypse store. These are the things that we are selling. I know it's a very limited inventory, but these are the really high sellers; these are the ones that I wanted to sell. So we have this product ID, our product name, price, and production cost. Then we have this apocalypse sales; this is how many sales we've actually made to our customers. So we have this customer ID, our customer name, product ID, order ID, unit sold, and the date it was purchased. And then we have our customer information right here. Here are all of our clients, so we have this customer ID, customer address, city, state, and zip code.

So now that we've taken a look at our data, let's go and load it into Power BI. So we're going to say import data from Excel; we're going to choose this model right here, and we're going to click open. And we are going to want all three of these, so I'm going to click on all of them, and we're just going to load it. We're not going to transform the data at all.

So now the data has been loaded. Let's go right over here on the left-hand side to our model tab, and let's scoot this over just a little bit and move back. And we're going to move these tables up to where it's a little bit easier to see. So right off the bat, you can already see that there are these lines between these tables, so there are already relationships that Power BI has automatically detected and created. From my experience, Power BI actually does a really good job at creating these relationships automatically, but we're going to go in and take a look at these and kind of see what everything means. And then we're going to go back and create these relationships from scratch just to make sure that we know how to do every single part.

To get us started, let's double-click on this line connecting the customer information table to the apocalypse sales table, and it's going to bring up this edit relationship page right here. So this line right here connecting these two tables actually gives us quite a bit of information without actually having to click into this edit relationship page. What this is showing is that we have a one-to-many relationship, and there's only one or a single crossfilter direction, and you can find both of those things right down here. And I'm going to walk through what those mean in just a little bit. On this page, you can also see the columns that Power BI decided to choose in order to tie these two tables together. Now, for our example, they decided to use the customer and customer right here from the customer information table, as well as the apocalypse sales. But I don't really want to use those specifically, specifically because on this apocalypse sales table, I might remove this customer information and just keep the customer ID. It may have chosen these customer columns because they have the exact same name and really the same information, but I want to use this customer ID anyways. So what I'm going to do is I'm going to click on that column and click on this column, and then I'm going to click okay. And if we go back into it by double-clicking again, we're going to see that. And now save that. And if we did what we just did before, which is kind of hover over it, it's going to show us what those two tables are joined on.

So opening this back up, let's go down here to this cardinality and cross-filter direction. Cardinality has several different options that you can choose from: you have one-to-many, one-to-one, one-to-many, and many-to-many. Now, for this example, we're looking at apocalypse sales, and we're going apocalypse sales down to customer information. Now, there are a lot of rows in the apocalypse sales, but there's very few in this customer information, and there's only one customer per row, whereas in the apocalypse sales up here, the customer can have several rows for several different orders. So that's why the cardinality is many-to-one. Now, if we flip this and we say we want the customer information here and we want the apocalypse sales down here, we tie that together; now it's going to flip, and it's going to say one-to-many.

Now let's look at the cross-filter direction, and there's only two options here: it's either single or both. And if we choose both and we click okay, this now goes from a single arrow pointing in one direction to two arrows pointing in both directions. But what does this really mean? So, in order to demonstrate this, I'm going to put this back to a single direction, and what we're going to try to do is connect the data over here or the columns over here to the columns in this apocalypse store. So let's go over here to build a visualization, and what we're going to do is we're going to take this customer information, and let's just say we want to look at state. So I'm going to click on state right here, and I'm just going to make this into a table. And the customer information table is only tied right now to the sales table, so we're actually going to go over to the apocalypse store, and we want to see how many product IDs are being bought in these different states. So really quickly, we're going to come up here and create a new measure, and all we're going to say is this measure is the count of apocalypse store product ID, and we're going to create that, and now we're going to select it, so it's added to that table. So now what this is showing is that there are 10 product IDs, which there are 10 products for each of these states, but that's not actually technically correct because not every state purchased these 10 different items. If we go back to our model and we change both of these to a both direction, and then we're going to go back and see what changed in our numbers. So now let's go back to our visualization, and now we can see that Minnesota actually only ordered seven different product IDs, Missouri 8, New York 9, and Texas 10. This is actually much, much more accurate than before. When you use the both option, it takes these tables and treats them as if they are a single table, but the single option is not going to do that. And so, for our example, if we're trying to connect this table to this table, and one of the last things that I want to show you is this option right down here which says make this relationship active. Now, if we don't click this and there are other options in here that connect these things like the customer to the customer, then that may be the active relationship. But if I select this is the active relationship, that means this is going to become the default relationship between these two tables. So now let's come out of here; we're going to click cancel; we're going to zoom in just a little bit and bring these tables a little bit closer, so we can zoom in just a little bit more. Now we are going to go ahead and delete these, so we're going to say delete, yes, and delete, yes.

So just for demonstration purposes, we're going to build these relationships from scratch. So we're going to come over to the customer information table, and we're going to drag it all, all the way over here and put it on top of this cust ID or the customer ID in Apocalypse sales, and it's going to automatically create that relationship. And we can open this up, and as you can see, it created the relationship between this customer ID in the apocalypse sales and the customer ID in the customer information. It also defaulted the cardinality from many-to-one and the cross-filter direction to single, so we're going to go ahead and change that to both and click okay. And then we're going to come over here to the product ID in Apocalypse store and drag this over the product ID in the apocalypse sales, and again, if we open it up, it created that relationship for us; it created the cardinality automatically, and we're going to change this cross-filter direction to both and click okay.

And so, on a really small scale, that is how it works. Of course, it becomes a little bit more complex the more tables that you add and the more relationships that are created, but this is how you're going to actually create the relationships in the model tab within Power BI. I hope that this tutorial has helped you understand this concept a little bit better. Thank you guys so much for watching; I really appreciate it. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

What's going on everybody? Welcome back to the Power BI tutorial series. Today we're going to be taking a look at DAX. [Music] Now DAX stands for Data Analysis Expressions, and it's basically a library of functions and operators that help you build formulas. You can use DAX to create measures and calculated columns within Power BI, which can really give you a lot of insight into your data. Honestly, it is not super complicated, and hopefully, by the end of this video, you'll have a lot more confidence actually using DAX and Power BI. So, without further ado, let's jump onto my screen and get started with the tutorial.

All right, so let's take a look at our tables and data before we get started. So we have two, two tables: the apocalypse sales, the apocalypse store. For this apocalypse sales table, we have the customer, product ID, order ID, unit sold, and the date it was purchased. And then for the apocalypse store, we have product ID, product name, price, and production cost. Now these are joined together, or they do have a relationship together via the product ID. So what we're going to be using are these new measures and new columns to create our DAX functions. So really quickly, let's, let's go over to this report tab and let's drop down our fields over here so we can see everything. And so, to get us started, we're going to go right up here to apocalypse sales; we're going to right-click and click new measure, and it's going to open up this right here, which is basically our bar where we can create our functions. And so right here, it's automatically given us the name measure, but we can change that, and we're going to say count of sales. So now we can start writing our DAX function; that's just going to be the name of it, and what's going to show up right over here once we click enter. So let's go over here, and we're going to say count, and as we're typing, it's automatically giving us options; it has something called Intellisense. If you've ever used other Microsoft products, Intellisense is their kind of autocompletion that helps you look at other options very quickly. And so we're just going to click on this count, and it's prompting us to put in a column name. And so we can come down here, and we can select one, or we can type it out, and it'll try to predict and help us choose which column to select. So for us, we're going to use this order ID, but let's just start typing it out; we'll say order ID, and then we can click on it, and we're going to close this parenthesis and click enter, or you can go over here and click this check mark, but we're just going to click enter. And so, over on this right side, it finalized that and saved that, and we can actually look at that by clicking on this box next to it, and we want to look at this in a table. So now we can see that there are 74 sales. Now, for this, we want to see who's buying our products; we want to see what our, what our client name is. So we're going to go over here; we're going to choose customer, and we're going to put customer on top of sales, and we're just going to take a look at it like this. So now we can see that our number one customer is Uncle Joe's Prep shop; he has 22 orders. Now they have the most orders with us, but it doesn't necessarily mean that they're spending the most money with us, but we can take a look at that later. The next thing that I want to take a look at is how many products we're actually selling; what are our big products that we're selling? We have 10 different items, but I don't know exactly which one is selling the best; if, if one is doing really poorly and getting no orders, this is something that I want to look into. So all we're going to do is go right back up here to apocalypse sales again, right-click and select new measure, and for this one, we're going to call it the sum of products sold, and all we're going to start out with is by doing sum. And if this seems familiar to something like Excel, you're 100% correct; it is very similar. And remember, these are both Microsoft products, so there's going to be similar functionality in both of them. And so this DAX is going to have a lot of similarities to exactly how it has it in Excel. So we're going to do an open bracket, and now what we're going to choose is this units sold; we want to sum up all of these units sold and see how many we actually selling. So we're going to say unit, it's sold; I'm going to hit tab; it's going to autocomplete that; I'm going to close my parenthesis and I'm going to come over here and click this checkbox. So now it's created that measure, and we're already selected in this table, so all we have to do is click the check mark, and it's going to show us that we have 3,000 total products sold, and we can go through here and see what the big sellers are, and probably the biggest one that I see right off the bat is this multi-tool survival knife. So these DAX functions that you can write can be very simple and lead to really good insights that you can use for the visualizations later on.

Now I want to take a look at the difference between something like sum, which is an aggregator function, and something like SUMX, which is an iterator function. Because if you add X to some of these aggregator functions, you can create them or, or make them into an iterator function. So you can have SUM and SUMX or average and AVERAGEX; adding X onto the end of them can make them into an iterator function. So let's take a look and see how that actually works. I'm going to show you the difference, and then I'm going to talk through the difference at the end. So really quickly, let's go back to our data and let's go to the apocalypse store. Now what we have right here is we have the price and we have the production cost, and we want to see how much profit we're getting from each of these, as well as we can take a look at the unit sold and see how much money we are actually making. So what we're going to do is we're going to come back over here; we're going to go to apocalypse store; we're going to right-click and create a measure, and in just a little bit, we're going to be creating a new column, and that'll kind of show the difference really well. So we're going to create this new measure, and we're going to name it profit, and we're going to come over here, and what we're going to do is we're going to take the sum, oops, we're going to start with our sums; we're going to take the sum of the price, and then we're going to close that parenthesis, and we're going to subtract the sum of the production cost. So all that does is it says if something cost $20, if we sold it for $20 and it only costs a $10, that's $10 in profit for that item. And then what we're going to want to do is we're going to actually want to encapsulate that really quickly because we're about to use multiply, and then we're going to sum, and now we're going to take the units sold. So how many units were actually sold at that profit that we just made? So let's see if that works, and let's click the check right here. And so we have the profit. So let's click on the profit; oops, that's not what I wanted to do; let's use a new one; let's create a new uh table; we're going to click profit and let's make it a table, and I'm going to pull this right over here. Now we have our profit, but I really want to know is which customer is spending the most money at my store. So we're going to come right over here; we're going to click on customer and customer at the top, and just at a glance, we can see that Uncle Joe's Prep shop is spending the most money at the store. Now what I want to show you is the difference between SUM and SUMX. So what I'm going to do, so I'm going to go back to this profit and going to copy this, this entire thing, and we're going to go back here to this table. Now we just created a measure, and we were able to break it down by each customer. So let's go back over here. Now let's go up here to home, and we're going to create a new column, and we're going to call this profit underscore column, and we're going to literally paste the exact same thing into here, and we're going to hit enter, and each row is the exact same thing. So what it's doing is it is going through the price and it's adding all of it up and calculating it at the bottom; it's adding the production cost; it's going all the way down and calculating it at the bottom, and then it's going over and looking at how many units it sold, and then it's performing this calculation up here, and then it gives us the total, and it's doing it for every single row. But that's not really what we wanted to show; what we wanted to show is the profit for each row. What we wanted to say is here's the price for the rope, the production cost for the rope, and then how many units we actually sold, and then it'll calculate that and give us the actual profit for just that row. But we cannot do it by just using this SUM; what we need to do is use something called SUMX. So let's add another column; let's go back to home, say new column, and now we're going to say profit_column_SUMX, and now we're going to use SUMX and hit Tab, and we need to choose the table that we want to put this in, so we're going to say apocalypse sales because that's the table that we're looking at right here; we're going to say comma, and now we need to input an expression, which it says it returns the sum of an expression evaluated for each row in a table. Before, when you're just using SUM, it's looking at all these combined; now it's taking it row by row. So what we're going to do is basically input the same thing as we did before; I'm going to copy; I'm going to paste that; it's not going to be correct; I need to get rid of these sums, but it's basically the exact same equation; give me just a second, and let's get rid of this sum, and let's see if this works. So let's click the check button, and now this looks a lot better. So what this is now showing us is at a row level; this nylon rope made us 51,000, almost $52,000; the waterproof matches made us $155,000, and we can go down and look at each item and see how much that actually made us versus this profit column. And so that is the biggest difference between SUM and SUMX. Hopefully that made sense; I know that SUMX and, and the difference between an aggregator function and iterator function can be a little bit confusing, especially if you've never done it before, but hopefully that was a good example for you to understand that concept.

Now let's go back over here to apocalypse sales. Right here we have a date purchase. Now in the DAX function, we have some ways that we can interact with dates, and so I want to take a look at those really quickly. So we're going to go right up here and click on new column, and we're just going to leave that as column, but what we're going to say is day. So there's a few different ones: we have DAY, DATESYTD, NEXTDAY, PREVIOUSDAY, and WEEKDAY, and they all are pretty self-explanatory. If you click on it, let's click on WEEKDAY; it says it's going to return a number from 1 to 7 identifying the day of the week of a date. So let's use this really quickly, and so we're going to say date purchase and click tab, hit comma, and

It's it's going to give us three different options, basically. It's a one, a two, and a three. Um, right here, if you hit this button, "Read more," you can read more on it. This is going to say Sunday is equal to one; Saturday is equal to seven. I like this one personally, which is Monday equals one. In my brain, it just makes more sense, so I'm going to click on two. I'm going to close that parenthesis, and we're going to—I guess I'll say—uh, let's say "Day of week" for the column. Let's click that checkbox. And now Saturdays are equal to sixes, Mondays are equal to one. This allows us to see which day of the week people are buying the most products on, or or which day of the week is somebody submitting their orders on.

And so let's go over to our report. Let's get rid of this. Just going to move this. Oh, jeez, I hate moving stuff sometimes. All right. Really quickly, I want to show you the difference between what we just did and what we already have. So we have this um, "Date Purchase," and let's make that into a bar graph. And what we're going to be taking a look at is actually the units sold. So right here we have this, and obviously for—we don't want 2022—we're going to get rid of the year. We only have one quarter right here. We can see January, February, March. So we can tell that January has the most sales, or the most units sold in that month. If we get rid of that, we go down to day. We do have some information, information, but we don't know what day of the week it is. It could change from month to month, and it's really hard to tell exactly what—if there's any pattern there at all. That's where what we just created comes in handy.

So let's recreate this exact same thing, but instead we're going to use "Day of week." So we're going to select "Day of week" and "Unit Sold." Let's drag that down, move this over right here, and this "Day of the week" should be on the x-axis. And it's really easy now to see if there's a pattern here. There's really not, at least not for this fake data that we have. Um, but just—I—I want these uh, data labels on really quickly. Um, it's not easy to see if there's any pattern again. Monday has the most, so maybe that—that—I mean, it goes down a little bit, and then it picks back up. So maybe middle of the week is our least uh, sales day. Our Wednesdays and Thursdays are a little bit lower than the rest, and the beginning and the end of the week tend to be the highest. Again, not a huge pattern, but but you know, it's much easier to see if there is a pattern from week to week, or what day of the week, now that we use this weekday function. And so this can be really, really useful.

Let's go back here to our data, and now we're going to look at our last DAX function for this video. Let's go up here and create a new column, and we're going to be looking at something called the IF statement. Now, if you've ever used Excel, I'm sure you have heard of this, and you can do the exact same thing here in Power BI. And so we're going to name this one "Order Size," order_size. And so all we're going to say is—if—we're going to click on this one right here—we need to perform our logical test, and then we want to say if it's true, what's our value, and if it's false, what is our value. So what we're going to be looking at is "Units Sold." So we're looking at order size, so we're going to say, "If 'Units Sold' is greater than 25," what's going to happen? If it is true, if the order is larger than 25, you want to say it's a big order, and if it's not, we want to say it's a small order. Super simple. We'll close that parenthesis, we'll click okay, and now really quickly we're able to see if this is a big order or a small order. And so that is all I have for you today. There are a lot of other DAX functions, but the ones that we looked at today are ones that are very common, ones that you'll see the most, and there can be a lot of really complex and intricate DAX functions that you can create. And in our project at the end of this series, I will be sure to include some more complex DAX functions, but hopefully this gave you a good introduction into DAX, so you know how to use it a little bit better. Thank you guys so much for watching. I really appreciate it. If you like this video, be sure to like and subscribe and check out all of my other videos on everything data analyst related. I will see you in the next video. [Music]

What's going on, everybody? Welcome back to the Power BI tutorial series. Today we're going to be looking at how to drill down and [Music] visualizations. So when I say "drill down," I mean you're basically adding another layer beneath the top layer of the visualization. And when somebody clicks or drills down into that data, they can see more insights and more information on the top-level data. When you drill down, you can also drill up, and I will show you how to do that in this tutorial. So without further ado, let's jump on my screen and get started with the tutorial.

All right, so before we get started, I wanted to remind you that you can find the data that we're going to be working with in this tutorial in the description. You can go and download it from my GitHub. Now, the two tables we're going to be looking at are "Apocalypse Sales" and "Purchase Tracker." And if you've ever created any visualizations, you've probably seen something like this, where you'll have the store and the price, and this is the—the things that we actually bought. So this is the toal total amount of Apocalypse prepping uh, equipment that we bought, and we'll put the store in this Legend right here, and you've probably seen something like this. And if you're anything like me, you're going to be in a meeting, and you're going to be presenting this, and some higher-up is going to be like, "Hey, Alex, that looks great, but I want to, you know, see what things we actually bought in Target. How much this cost? Can you create a visualization for that?" And you're going to be like, "Well, I could, or I could use drill down." And so you could have done this in the first place, uh, which you should have. So what we're going to do is—all we're going to do is—we're going to say—we're going to say the product right here, and these are going to be the actual things, and we're going to put it right under "Store." Now you can't see these things right, but there is a a hierarchy here. So once we added this, these options became available. Let's take it out, and all those just disappeared. And then if we add it back right here, they came back. And so you can do right here, which is "Click to turn on drill down." You can go to the next level in the hierarchy, or you can even "Expand all down one level in the hierarchy." So let's look at each of those really quickly. So let's click on this one. It's just going to turn on drill-down mode. So now if I go and I click on Target, it's going to drill down into these, and if we want to, I can then put "Product" under this Legend, and we can see all of those things. But of course, if we go back up, it's going to be all broken up into this clustered column chart, which is more like um, this, which isn't exactly what we were going for, but it worked, works now. Uh, let me get rid of this. I actually want "Store" in the legend. Now if we turn that off and we click it, it doesn't do that anymore. So what it does now is it just highlights Walmart, it highlights Costco, it highlights Target. So we're going to keep that on, uh, but we can also do something called going down the next level of hierarchy. So let's click on that, and so now this is going to go down to the next level, down to this product level, because that is the next level. And now it's going to show us each of those things, but it's going to have it broken out by the store. And so it's a completely different visualization, but all within the same realm of the data that we're looking at and what we actually care about. So let's go back up in the hierarchy, and then let's use this one right here, which is "Expand all down one level in the hierarchy." And so this one is again extremely similar, except it just visualizes it differently. And now what it's doing is Walmart rice, Target dried beans, Costco rice. So instead of having an all uh, like this one where it's stacked on top of each other, it's breaking it down individually. So this one column would become three separate columns. Now I'm going to minimize this right here. Uh, I'm actually going to go back up in the hierarchy just for visual purposes. Now I'm going to show you one more example. We're going to use this "Apocalypse Sales" up here, and this is one that I actually use all the time. So the one you've seen—you know, you'll get stuff like that, especially if you're working with like sales and stuff, but I work in operations, right? So I have a lot of order IDs, product ID, stuff like that. Now this one, this one genuinely I use quite often. I'll have a customer—U—let's make it—we'll just go like this. We have a customer, and we have "Unit Sold," and let's use the customer as the legend. So let's make this one quite a bit larger, and I'll have something like this, and they'll say, "Okay, well, we want to see the order IDs that go with it, because we want to know what orders are actually happening for each of these people." Obviously, I'm not using this exact data, but very, very, very similar. And all you have to do is take these order IDs and slide it right under here, under "Customer," and this visualization right here is something I've done a thousand times, because what happens is is someone—some stakeholder in our company is saying, "Hey, Alex, we want this, and we want to know—we want to drill down on this IP address. We want to drill down on this certain database. We want to drill down on something, and we want to see the order IDs within them." So then all you do is you turn on drill mode or drill-down mode, you'll click on it, and you can see every single order ID that's in there, and then they can go and look those up in their system and resolve them or whatever they're trying to do with it, and it helps a ton, and it's very, very useful. This one is extremely applicable, and that's really all drill down is. Again, you have these different hierarchies as well, um, but for different things, it's not as useful as you can see. We also have this hierarchy, which again is not as useful, so it just depends on the data that you're using and how you want to use this drill-down effect. But I promise you that drill down is used all the time, especially when you're giving presentations where people want to know more information than just the—the visualization that you're presenting. So I hope that this has been helpful. I hope that you understand drill down a little bit better. If you like this video, be sure to like and subscribe and check out all my other videos on Power BI. Thank you, and I'll see you in the next video. [Music]

What's going on, everybody? Welcome back to the Power BI tutorial series. Today we're going to be taking a look at conditional [Music] formatting. Now, conditional formatting may sound familiar because we looked at it in the Excel series, and it's very similar how you use it in Excel versus how you use it in Power BI. Conditional formatting allows you to take a table or a matrix within Power BI and use those cells to color-code them and create gradients and different visualizations within the actual table or matrix. I'm excited to start this one, so let's jump over my screen and get started with the tutorial.

All right, so before we get started, if you want to use the data that we're using in this video, you can find it in the description on my GitHub. Now, conditional formatting is super simple, and you've most likely used it in Excel before, but you can also use it in Power BI, and let me show you how to do that. So the first thing we're going to do is come over to our "Apocalypse Store," and we're going to pull up our "Product Name" as well as the "Price," and what we can do is come over here, and we're going to go to "Price," and it has to be under the columns. So you can't come over here and do this. We're going to come right over here to "Price," and we're going to right-click and let's go to "Conditional Formatting." And we have "Background color," "Font color," "Icons," and "Web URL." Let's take a look at "Background color" first. This is most likely the one that we'll look at the most. So we're going to get this popup, and I'm going to slide this over. Now there's a lot of different things being customized in here, and the first thing I want to take a look at is "Format style." We have the gradient, and what it's going to say is the lowest value will be this color, highest value will be this color; it'll give us this gradient color scale. And so we'll use that in just a little bit, but we can also create rules, kind of like an IF statement, and if it is between this range and this range, we'll give it a color, and if it's between a different range and a different range, we'll give it a different color. So we'll also try that one, and then we have this "Field value," uh, and this one is one that—honestly, I don't use that much. I've used it maybe once, and what you can do is select a text field like "Customer," and you can do some summarizations on the first and last, and that is it. So what we're going to do is we're going to look at "Gradient," specifically for—not the customer—but we're going to go back to the "Apocalypse Store," and we're going to do it on the "Price." Now what I'm going to do is keep it as the count because this is what the default is, and we're going to go back and fix it later, but what we want our lowest value to be is this bright green, showing that this—it's a cheap product; it's easy to purchase. The high-value ones are going to be just the shade of red—more expensive—and we'll do it on the count. Now remember, the count is on each of these, and we're not doing a count of how many are sold; we're doing a count of each product, so it's just one per row, so it all should be the same color. Let's take a look. So it is all the same color, but what we really want to show is the actual price, not just the count of the price. So let's go back to "Conditional Format Adding," we're going to click the "Background color" again, and this time we're going to change the summarization. Now you can do sum, you can do average, minimum, maximum; it really doesn't matter for this example; the number is the same regardless of really which one we choose, so we can just choose the minimum, and it's going to choose the minimum of each row, which is the price, so we're just going to select minimum for this example. We'll select okay, and it should correct it accordingly, which means the bright green is the lowest, and it goes all the way up to the highest, which is the red.

Now let's go over here to "Apocalypse Sales," we'll add in the "Units Sold," and let's move that out a little bit, and I'm doing that on purpose because we're about to look at something within the conditional formatting. So let's go to "Unit Sold," and we'll look at the conditional formatting for this one. Now, if you noticed, we now have a new one on here called "Data bars." Now we're able to see "Data bars" on "Unit Sold" and not "Price," because "Unit Sold" is something like a sum, an average, something that's aggregated. But let's take a look at "Data bars" because I want to show you how to use this, and then we'll go back to the "Background color." So for "Data bars," we are going to be taking a look at the lowest to the highest value again. We're going to go from bright green all the way to this exact red; it's going to be from left to right, and what it's going to show you is if it is a positive number, which all of these are, is going to be a green bar, basically representing the number that you see in here along this line. So let's click okay, and we're going to be able to see the highest numbers, and let's scooch this over quite a bit so you can kind of get a better understanding, and we're going to do it from highest to lowest. So we sold the most multi-tool survival knives at 477, and so this entire bar, this row, is entirely filled up, or almost all the way filled up, while as it gets lower, and as we sell only 182 solar battery flashlights, the bar is going to represent that and show that.

Now I'm about to completely mess up this visualization on purpose because it's about to get very messy to show you that you can do a little bit too much. Uh, it is possible. What we're going to do is we're going to go right over here to this "Background color," "Unit Sold," and instead of "Gradient," let's look at "Rules." Now with the price, we just did a gradient scale, but we can do basically groups of these and say if a number is greater to or equal than this number, then it's going to be a certain color, and then if it's in a different range, we can give it a different color. So we're going to say if it's greater than or equal to zero, and we're going to say "Number," not percent, and if it's less than 266, because we have 265 right here, let's make it a nice uh, like gold, a beautiful lovely mustard gold, just—just great. Now we're going to say if it's greater than or equal to—we'll do 266, because this is less than 266, so it should be greater than or equal to 266, "Number," and if it is less than—we'll say 500—now we want to do this one, and we'll give it uh, let's do like a peach, and we'll click okay, and now we have another conditional formatting on top of that that can give us more information. Now again, you should not do this; it's just too many. Now let's go one step further and make it even more ridiculous and show you one more thing before I show you how you may actually want to use this. Uh, let's go back to "Unit Sold," we're going to right-click, go to "Conditional Formatting," and you can do something called "Icons." Um, "Font color" is the exact same thing as "Background color," except it changes the—the font, and so I'm not really going to look into that one. "Icons" are very simple, extremely similar to Excel and how you've seen them, and the rules that you can apply to them are basically the same as if you're doing like a gradient, and it's these IF statements that we saw before. Now it autog gives us this right here, which basically says 0 to 33%, 33 to 67, 67 to 100. If it's in the bottom 3%, it gives us this red; the middle is yellow, and the top is green. So we can go through and change all of this, but honestly, this looks pretty good, so let's click on it, and so the ones that are least sellers are these red ones right here, and the top sellers are up here. Now this is just based on "Unit Sold," and this looks absolutely terrible. So let's kind of take this exact information but make it a little bit better. So we're going to create a new visualization, or at least a new table. So let's click on "Product Name," and we'll take the "Price," "Unit Sold," and "Revenue," and what I think makes the most sense for looking at "Revenue" is these "Data bars" right here, but there's only one problem: I can't do that because it's not summarized like "Unit Sold" was. But what I can do is to get that—those "Data bars"—is I can come right down here, instead of saying "Don't summarize," I can summarize it, and I can just click the—so it now is summarized; it's the exact same number.

But if I right-click on "Sum of Revenue," go to "Conditional Formatting," I can now use those data bars. So we're going to use those data bars, and we're going to say for the lowest value and the highest value, and let's just make it a nice, maybe a darker green. I don't want it to—well, that's that's hideous—let's make it this color right here, a nice dark green. And there's no negative, so it doesn't really matter. We're going to go left to right, and you can show the bar only, but we're going to keep it because I want to see it. And we're going to go just like this. We're going to order, and this is pretty telling. Um, honestly, I did not think the weatherproof jackets were performing so well, but I mean, they are by far a number one seller. So you know, our weatherproof jackets, multi-tool survival knives, and the nylon rope are outperforming all of our other products. So those might be the ones that I focus on the most, while duct tape, the N95 masks, and waterproof matches—I mean, those are those are garbage—so I might be looking to replace those in the near future with some other items that might sell a little bit better.

So that's how you use conditional formatting, and it's actually pretty useful. There are a lot of times where I've done something like this in an actual visualization for work, and it looks something like this. It just depends on what you're visualizing, but this is very much a simple thing that you can do to just add a little bit more information and actual visuals to this little chart or table that you're going to create. Sometimes it's just better to have these simple visualizations on this table rather than just having the numbers themselves. It makes it a little bit more easy to read and understand.

So again, I hope that this was helpful. Thank you guys so much for watching. I really appreciate it. If you like this video, be sure to like and subscribe and check out all my other videos on Power BI, and I'll see you in the next video. [Music]

What's going on, everybody? Welcome back to the Power BI tutorial series. Today we're going to be taking a look at bins and [Music] lists. Now, bins and lists are really useful because they allow you to group things together to analyze and visualize them easier. So in this tutorial, I'll show you how to create your bins and lists, and then we'll create some visualizations to show you how it can be helpful. So without further ado, let's jump on my screen and get started with a tutorial.

All right, so before we get started, I wanted to let you know you can go and download the data that we're going to be using in this tutorial in the description below. It's on my GitHub. So we are going to be looking at bins and lists today, um, and for this we're going to be going over here to this "Apocalypse Sales," uh, and let's open up our data right over here. And we want to look at "Apocalypse Sales" really quickly. I feel like more people would know what a bin is, so we'll kind of start with a list, just go a little bit backwards than we normally would. Uh, I'm going to use this "Customer," or we're going to use this "Customer" column right here for a list really quickly. And you can do that in two ways: You can come up here and you can right-click on the "Customer" and go to "New Group," or you can come over here under this, uh, the "Field" section on the far right and go to "Customer," right-click, and click "New Group." So let's click on that now. And right now it's only giving us the "List" type; it's not giving us "Bins" because bins have to be numeric, so we really can't do that at the moment. Um, so we're going to call this just "Customer Group," or we'll actually call it "List" just so it's easier to recognize when we create it. And so all we're going to do is we're going to basically group these, but it's going to be called a list. And so what we're going to do is we're going to select, and we're going to select, and we're going to say "Group" and click on this "Group" button, and then it creates this "Alex the Analyst," "Apocalypse Preppers," and uh, this "Prep for Anything" prepping store. So that it kind of named it for us, but if we double-click on it, then we can rename this, and we can call this "The Best Prepping Stores." And then we have these last two, and we can we can click on one and then click "Control" and click on the other one, so we get both of them, and then we can click "Group," and we can call this—and we'll double-click—and we'll call this "The Worst Prepping Stores." Um, and then that's it, and that's we have to do. And what we're then going to do—and if you want to undo this and you want to switch it up and do whatever, you can click "Ungroup," but we're not going to do that. We're going to click "OK," and here is the column that it created, and it basically tells us what list we put it in. If it's "Uncle Joe's Prep Shop," that's in the "Worst Prepping Stores" list, and if it's the "Alex the Analyst Apocalypse Preppers," that is in the "Best Prepping Stores." So it's kind of like an "if" statement. You could even create a calculated column, do it on this "Customer," create an "if" statement. This is just a lot faster and a lot easier than doing that, but it basically would do the exact same thing.

Now you can use lists as well on things like numeric. So let's say we have "Order ID," and we'll go to "New Group," and it's going to auto-go to "Bin" because typically that's what you'll use, but you can do "List" as well. And let's say, you know, we want to say we want to call these like—we'll group these and call these "The First," um, we'll call this "The First Customers" or "The First Orders" because we're looking at order IDs, look at the first orders, and then we will go back here. We're on the left side; we're going to click—oops—we're going to go back to the top; we're going to hit "Shift," group all of these, and we'll say "The Latest Orders." And you absolutely can do this. Um, again, this is kind of like an "if" statement, right? So you're saying if it falls between this range and this range, then it's called "The First Orders," and if it's between this range and this other range, it's "The Latest Orders." Um, again, it's just a much simpler version of an "if" statement, and so you don't have to write it all out; you can just have this user interface kind of do it for you, uh, and and it's really, really useful.

So now let's talk about bins, and by far the easiest way to demonstrate this—and I'll show you one other way—but by far the easiest way to show this is by using "Age." And so, uh, for absolutely no reason whatsoever, these customer IDs, uh, who are right here in this "Customer Information," they decided to give us some of their buyer information who are actually buying their products on their website or in their store; they just decided to give it to us as well as some, uh, simple demographic information. I I don't know why, but what we're going to use bins for is grouping these age brackets. So you know, you might be interested in, say, "Well, I want to know if my core population who are buying my products are within a certain range," and you don't want to look at every single age because then it just, you know, on your visual visualizations it's not going to look right; you want to kind of group them and make it easier to visualize. So what we're going to do is we're going to go through here and we're going to basically go by 10: 10, 20, 30, 40, 50, 60, and see what age bracket these people fall in. So we're going to go to "Age," we're going to right-click, and we're going to say "New Group," and we're going to go to "Bin," and we'll leave it as the default "Age Bins," um, and you can do two things: You can do the size of the bins, which splits it, uh, uh, which splits it by this number right here, or you can go based on the number of bins. So if you only want to do five different bins, it'll calculate that for you, and it'll say, "Okay, if you only want five bins, you're going to have to do it at 12.2. If you want 10 bins, it can be 6.1," but it is completely up to you on how you want to do that. Um, you can do the size, and we'll just say every 10, which is what we're going to do, or you can go through, and then you can create, you know, the how many bins you actually want. So let's go ahead and click "OK," "OK," and it's going to create those bins for us. So if somebody is 78, they're going to be in the 70s bin; if somebody's 41, they'll be in the 40 bin; if somebody is 29, they'll be in the 20 bin, and so on and so forth. So when we go to visualize this, we don't have, you know, 71, 72, 73, 74, have a lot more things on our visualization; it'll just be the 70, or it'll just be the 20.

Now we can also use bins on dates as well. So let's go back to "Apocalypse Sales"; we have this "Date Purchased," so we can create a bin for this as well. So let's go to "Date Purchased," let's go "New Group." Now you can also create a list, and that's totally fine if you would like to do that, um, and it would look kind of like this where you can go through and you can select it and you can say, "Okay, this group all these dates," you can group those and say this is going to be January, uh, and you can do that, and that's totally okay, um, but for this one we're going to do bins. I think it's a little bit easier to do bins because what we can do is go right here and we can specify what we want: seconds, minutes, hours, days, months, or years. And so, um, for the data that we have, it goes January, February, and March, so we're going to do months, and we're going to say the bin size is going to be one month, so each month should have its own bin, so it'll be three bins total. So we're going to select "OK," and as you can see on this right side, we have January of 2022, and that correlates to the January over here, then it goes down to February, and then it goes down to March. And then when we visualize this, uh, we don't have to do this the hierarchy stuff that we do in here where we filter it down down to months; we can just use this right here, and that will be our months column. So now let's go over to our visualizations, and we'll see how this looks really quickly. We're not going to look at all of them, but we will take a look at a few of them. So the first one that we can look at is "Age." So let's look at the "Buyer ID," and then we'll do "Age" as well. And so let's spread this out, and we can see our distribution of our buyers. So it looks like we have very few, uh, who are in the 10 range, thank goodness, and we can even put the "Age" right under here under the "Age Bins," and we have this. Now we kind of have this drill-down, and so if we go right here and we drill down right there, this will actually give us the breakdown. So this is what it would have kind of looked like our visualization would have looked like if we had just kept it the "Age," because now we're drilling down into to the "Age," and so it looks like we have one 18-year-old and maybe a 20-year-old as well. Um, let's go back up. Yeah, so it looks like we only have one "Buyer ID," yes, so there's only one 18-year-old, so of legal age to start buying, you know, all these prepping equipment and probably, uh, buying online and stuff like that, which makes sense, right? So, uh, this gives you kind of a quick breakdown in the bins rather than um doing it the alternative way. So now let's take a look at the "Customer List" as well as the "Unit Sold," and it looks like the "Best Prepping Store," uh, is actually performing much worse, surprisingly, uh, than the "Worst Prepping Store." And so I hope this gave you a really good idea of how to use bins and lists within Power BI. Thank you so much for watching. If you like this video, be sure to like and subscribe and check out all my other videos on Power BI. I'll see you in the next video. [Music]

What's going on, everybody? Welcome back to the Power BI tutorial series. Today we're going to be taking a look at all types of [Music] visualizations. Now, when you're working in Power BI, there are a lot of different options to create visualizations, and you may not always be sure which one to use, and so that's what this video is for. I'm going to walk you through a lot of the visualizations that I like and I use a lot, as well as kind of point out some of the ones that I don't like as much, so that you get kind of a feel for the ones that I think are really popular and that are used the most. So without further ado, let's jump in Power BI and start taking a look.

All right, before we jump into it, there is a link in the description where you can get the data that we're going to be using for these visualizations if you want to practice them yourself. Before we actually get into it, we do need to combine this, and if you download that Excel and you see this, you'll have to do the same thing. All we have to say is that this "Product ID" is the same as this "Product ID Purchase," and now we are good to go. Do one to many, and it's okay if it's one away. So right over here under this "Visualizations" tab, there are lots of different options, and it can be a little bit overwhelming; you don't really know which one to choose. There are some in here that I have almost never used for my job ever, so I'll point those out as we go through, but the main focus is going to be focusing on the ones that I do use, that I have used, and showing you how to actually create that visualization, maybe spice it up just a little bit. But we have a lot of them to go through, so let's jump right into it. And the very first one that we're going to start with, probably the easiest one and the one that you'll recognize the most, is a stacked bar chart. And what we're going to do is go ahead right over here to the "Product Name," and we want this "Unit Sold" as well. So we're going to click "Product Name," and it's going to go straight into the Y-axis for us, and then we're going to click "Unit Sold," and that will go into the X-axis automatically; it just kind of intuitively knows, but sometimes it will make a mistake, and then you can just fix it or flip it. And we do want this—let me make this much larger—we do want this to be a little bit more color-coded; that is what this "Legend" is down here. So what we're going to do is drag this "Product Name" down to the legend, and now we have each product as its own color. And in previous videos, we have gone through and looked at some of these visual and general options that you have when you're actually creating these visualizations, but we're going to do some of them while we're in here as well. So we're just going to go down here, going to choose "Data Labels," and we're going to shrink that. And if you go higher, the higher you go, the less you see. So if you want all of them all the way down to the green, we're going to go right about there, and we're going to make it smaller. So now we can go ahead and click anywhere outside of that visualization, and now we can create a new one. If we had just kept it like this where we were still interacting with this visualization and we clicked on a different one, it would have then change our visualization completely, which we don't want. So let's hit Ctrl+Z, click out of it, and now we can create a new one. Let's go right over here to this 100% stacked column chart. I'm going to click on it, drag it over here, and make it much larger. And we're going to come right over here to this "Customer Information," and we're going to click on "Customer," and then we're going to go up to "Unit Sold" and click on "Unit Sold," and we want to break these out. And so basically what this is doing is it's breaking it out by each of these shops, and we can see the total of what they're buying, the "Unit Sold," but we want to see exactly what products make up this percentage or this 100%. So we're going to go right over here to "Product Name," we're going to drag that down to the legend, and as you can see now we have each of these products, and each of the products is up here. So this "Backpack," we can see the "Backpack" right here, "Backpack" right here and right here, and we can see which customer is buying what percentage of their purchases. So for this "Prep for Anything Prepping Store," they have a very large percentage; 40% is duct tape, so they're buying a lot of duct tape. So really quickly we're able to see what clients are purchasing or which clients are purchasing what products the most. So just like this "Alex Analyst Apocalypse Preppers," they're buying a lot of water purifiers. We like drinking clean water, um, you know, that's just what my audience likes, and so, you know, we can easily get a quick glance of that. Again, we're going to go in here; I tend to like putting these data labels on here; that's just what I preference, so, you know, something like this, it looks nice, it looks clean. Um, we can always go back and change these names, which we'll do for this one. So we're going to go over here, go to "Title," we'll go down to the text, and we'll do "Customers," oops, "Customer Purchase," oh jeez, "Breakdown"—pretend I'm really good at spelling—and we're going to do it just like that. We'll get out of there. So now we have "Customer Purchase Breakdown," and that looks really nice; it's a good, uh, a good visualization, and we're going to bring that right over here. We're going to have a lot on the screen, so I may have to, uh, make them smaller or larger to fit everything.

All right, so let's go on to our next one. Another really common visualization is this one right here, which is the line chart, and the line chart is great, especially when you're using things like dates. I have found this one to be the best, and a lot of people use this as well. So we're going to go right over here and click on "Date Purchased," and then "Unit Sold," and on the X-axis you can see it's broken up by year, quarter, month, and day. So we don't want to do it that high level; we only have three months of data in here, so we're going to get rid of the year, we're going to get rid of the quarter, and then we at least have this. And let's break it out because right now we're looking at all of the units sold, so we're going to drag the "Product Name" right down here to the legend, and now it breaks it out by the actual product, and for each month in January, February, or March, you can follow these products and see how they did in each of those months. And if we wanted to, we can come right over here to the filter on the "Product Name," and we could filter it by maybe the top three. So let's do "Multi-tool Survival Knife," the "Nylon Rope," and the "Duct Tape." We can have it just like this, and you know, you can do those for any product that you want, but again, we just want to do it for those three just for an example. And that really doesn't give us a ton of information; we could even go down to the day, and you know, it might give us a little bit more information, and so we'll keep it like that. And we can go over here, change the name as well. We're not going to do this for all of them; again, we're just looking at the different types of visualizations I think are really good to know, but we'll change this one as well to "Products Purchased by"

Date: We'll keep it just like that again. Nothing fancy; we're just trying to look at a bunch of different stuff. So let's put this over here, down here. Now let's click out of there. And there are other ones in here, um, that are definitely useful, and you absolutely can use, um, like this one is a stocked bar chart. This one is a stacked column chart; it's basically the same thing, just a different orientation. Like we went to here, it's just a different orientation; it's the same thing, um, just like this clustered bar chart, cluster column chart; it's just its orientation, either horizontal or vertical. Then we have things like an area chart, a stacked area chart—not really things that I've used too much in previous positions. One that I have used, though, is a line and clustered column chart. So it kind of combines a few of these with, you know, you have these bar charts as well as line charts into one visualization.

So let's look at this one because this is one that I have used several times in my actual job. So for our x-axis, we'll use the product name. Then we'll look at something like the price. And so let's make this a lot larger so you can actually see it. So now we have the price, and now we can look at something like the production cost, and that can be our line Y-axis. So now we're looking at the price of it, how much someone is actually paying for it, and then we're looking at how much it's costing us to actually produce that product. And so really quickly, at a glance, you can kind of see that it's around the halfway to 2/3 point on most of these. You can see that the production cost is always lower than the actual price because if we're out here to make a profit on these products.

So let's minimize this one; we're going to put this one right down here. Let's make it even smaller. Let's click out of that. And the next one that we're going to take a look at is a scatter chart. So let's click on that and make it much larger. Oops, there we go. So let's use the price and the production cost again. And so our x-axis is the price; our y-axis is the production cost, but now we need to fill in this values right here. So let's go over here and click on the product name and drag that into values. And so now we have our values; we just don't know what they are, but we can see it. So let's drag this down to Legend as well, and it breaks it out, and we kind of have this scatter plot. And, you know, for this fake data that we're using, it doesn't really show a lot, but if you're using real data, you can definitely find outliers and trends and patterns using this type of visualization. Let's go ahead and make that one small as well, try it right down into the corner.

Now let's go right over here, and we have the, the dreaded pie charts, um, and donut chart. Now, look, I think it's kind of a joke in the data analyst community about pie charts and donut charts, but at the same time, people use them, and they request them. And so sometimes you're going to use it whether you like it or not. So let's click on the donut chart, and let's make this one a lot larger. And let's go over here, and let's click on State, and we're also going to click on total purchase. And that's really all you have to do; these ones are pretty straightforward. You can change a few different things, like where these labels are; if you want them inside, you can also do that, and that would look totally fine. Um, again, I'm just not a super huge fan, but you will get this one requested; people like this and want to see it. And the reason a lot of analysts don't like using this is because when you start glancing at these, it's really hard to tell the difference between these sizes. If you look at something like this, you can easily see that this is larger; like if you're looking at this one, the multi-tool survival knife is obviously the longest, and it gets shorter, shorter, shorter, shorter. But when you start getting in here, it's really hard to approximate the size. I would not be able to tell the difference between this 5.63, 5.78, 2, 7.72. I would not be able to tell really the difference between these or, or kind of the, the difference between them very easily. That's why a lot of people don't want to use them in general. So again, I want to show you this one because I think it's worth noting and worth knowing how to use, but I don't really push people towards this because I don't think it's the best visualization available most of the time.

All right, the next two are super easy but are used all the time, uh, maybe more than some of these even, but they're just so easy to use, so I'm kind of saved them for last. This one is the card, and all the card is is it displays one number or multiple numbers if you want to use a multi-card, but we'll just look at the card for now. All we're going to look at is the total purchased, and it's just going to display it just like this, and you can make it as large or as small as you'd like. And normally it goes on like the top, and you'll put card here, card here, um, just for example, I'll kind of show you how this might look. So it'll look something like this, right? And at the top, it'll have different, usually high overarching information, and this is super common to see, and I'm sure if you've looked at other people's visualizations, you'll see something like this. This is usually totals or averages or something like that in here where it's super easy to look at. So like right here, this is total purchased, and we can go in and look at the minimum, and then we can go over here, and this one can be account, and so it gives us a lot of information just at a really quick glance. And then we have all of our more in-depth, colorful visualizations that kind of have more information than just a single piece like the card does.

And then the very last one that I'm going to show you is this one right here, which is the table, and this one is obviously extremely popular; it's like a little Excel table. And we can go in here, and we can get the customer, wherever that is, and then we'll also get the unit sold, and this is what it looks like, and it's super easy, and often times you'll have it like on the side as well, uh, and all the other visualizations over here. And so, you know, if we're going to take all these visualizations and pretend they were like a real thing, you know, there's a lot in here, but we'll just kind of really quickly do this, um, you know, we might have something like this, and we'll make this larger, make this wider, and, you know, we have a lot of information just in here. And this is not a project, so don't go put this on your portfolio; I'm just throwing a ton of random visualizations on, you know, this dashboard. But you can already see a lot of these; you most likely have seen in other people's work, in other people's visualizations on LinkedIn or on YouTube. These are very common, very, very popular. And again, we did not go through all of the ones over here; there are maps that you can use, but I haven't used maps ever in my job. There are things like gauges and decomposition trees and waterfall charts and, uh, tree maps and all these different things, but I really have never used those in my actual job, and I don't see them a lot in other people's work either. Otherwise, I would be telling you to learn these and use these. But again, try them out; see which ones you like. If you like this video, be sure to like and subscribe below and go check out all the other Power BI tutorial videos that I have on my channel, and I will see you in the next [Music] video.

What's going on, everybody? Welcome back to the Power BI tutorial series. Today we're going to be working on our final [Music] project. Now, this is our final project, the Power BI tutorial series. So if you have not watched all of those videos leading up to this, I recommend going and watching those videos so you can make sure that you know all the things we're going to be looking at in today's project. I am really excited to work on this project with you because I think it is a really good one, and it uses real data that we collected about a month ago where I took a survey of data professionals, and this is the raw data that we're going to be looking at. And so I think it's just really interesting that we collected our own data; now we're using it for a project. We're going to transform the data using Power Query, and then we're actually create the visualizations and finalize the dashboards as well as create a theme and a different color scheme to kind of make it a little bit more unique. Without further ado, let's jump onto my screen and get started with the project.

All right, so before we jump into it, I wanted to let you know that you can get the data below; it is on my GitHub. You can go and download this exact file that we're going to be looking at. Now, in the past several projects, we have been using this fake apocalypse data set; you know, it was fun, it was, you know, whatever. This data set is real; this is a real data set. It was a survey that I took from data professionals. I posted on LinkedIn and Twitter and all these other places, and we had about 600, 700 people who responded to the questions. So before we actually get into it and start cleaning the data and doing all this stuff in Power BI, I just wanted to show you the data.

All right, so this is the CSV that I downloaded from the survey website that I used, and this is completely raw data; I haven't done anything to it at all. Let's go through the data really quickly, and we'll kind of see what we have, and we are not going to make any changes at all in Excel; we're going to do all of our Transformations, or at least a few transformations, in Power BI because, again, this is a Power BI tutorial and project, so I want you to kind of learn how to use that and not use Excel because you can go through my Excel tutorial if you want to do that. So let's just look at it in Excel, and then we'll move it over to Power BI and actually start transforming the data. So we have this unique ID; these are all the people that actually took it. Oops, don't want to do that. We have an email, which this was completely anonymous; I didn't collect any data or user data on this. Then we have the date taken, um, and let's get into the actual good information. Then we have all of these questions. So we have question one, which title fits you best, and they can choose things. Now, uh, let's add a filter really quickly that we can look at this. Now you had the pre-selected ones, which were like data analyst, architect, engineer, but then there was an option where you could say other and you could specify what that was. So if you look in here, we're going to have all these different other, please specify, with different titles, right? And there were a lot of them. Now, typically what you want to do is really clean this up, and we're not going to be doing a ton, ton, ton of data cleaning, but we are going to do some in Power BI, but none in here. But typically with this amount of data and the way that it's formatted, we would do so much data cleaning. Um, with this one, I mean, there was a lot of work to be done. Um, like this current year salary, this is one that I would absolutely be cleaning up because it's ranges, and it has a dash and a k and all these numbers. This is something that I would be cleaning up and using, but we're not going to be cleaning this up right now. So anyways, let's just get into it; let's see what questions we asked. Uh, we have the yearly salary, what industry do you work in, favorite programming language. Then there were a lot of different options; this is like one question where they picked multiple options. So is how happy are you in your current position with the following? You have your salary, work-life balance, um, then we have co-workers, management, upward mobility, learning new things, um, and they could rank it from zero to 10. So some people ranked upward mobility a 10; some I'm ranked at a zero or a one, um, and again, they can answer however they want. How difficult was it to break into data? Very difficult, very easy. Um, if you're looking for a new job, we have, you know, what would you be looking for? Remote work, better salary, etc. We have male, female, which country are you from? And then this is more like demographics. So if you're a male, how old you are, and this was in a range. So this is like a, a, a, a sliding bar, so you could slide it to the exact age you had. There's some people who are apparently 92, um, which if that's true, I mean, good for you, man or woman. Actually, really quickly, I'm going to see just, just while we're here, I'm going to see if this is a male or male or a female. Oh, it's a female from India. Very cool. Um, so we have all this information, and it is a lot of information. When you have something like this, I mean, there is so much data cleaning that can be done. I mean, I already see like 20-plus different things that I would need to do to make this a lot better, um, and we also have date taken and the time taken as well as how long they took on it, like the time spent. Really just really interesting data, but again, this is a beginner tutorial series; this is the beginner project, so we're not going to get do anything too crazy. I will be using this exact data set in a future video, doing a lot more data cleaning and creating a much more advanced visualization with what we have and what we're looking at right here. But for this video, we're just going to be doing a pretty simple visualization and dashboard that you can use, uh, to practice with or put on your portfolio if you know that's where you're at right now. So let's get out of here and let's put this into Power BI.

So let's exit out and let's come right over here to import data from Excel. We'll click on Power BI final project and open. Give that a second; doing this all in real time. We only have the one, so we'll do be; we won't be practicing any joins or anything, but we're not going to load it; we're going to transform this data. So let's put it into Power Query editor. And now we have all of our data in here, and it should look extremely familiar. Now, when I'm looking at this, when I start looking at this information, I kind of need to know beforehand what I want to get out of this. Do I need to clean every single column? Do I just need to clean a few of them? Do I need to get rid of columns? That's kind of where my head's at. And so right off the bat, I can already tell you that there are columns that we can just delete to get out of our way. So we're going to do that at the beginning so that we don't have to to do that later on or they're just in our way. So I'm going to click on browser, and then I'm going to hit shift, and I'm going to go over here to refer, and I'm just going to go up here to remove columns. And everything that we do is going to go over here to this applied steps. If you've been following this series, um, you know, we can remove things, add things, but anything we do will show up right over here, so we can track it and go back if we need to. Now, one column that I know for sure that I'm going to be using quite a bit is this, which title fits you best in your current role, because I, I specifically wanted to do a breakdown of different people's roles and how much they make and different stuff like that. So I know that I want to use this, but as we saw before, there's kind of the issue is is it's not very clean, right? It has data analyst, data architect, engineer, scientist, database developer, and then like a hundred different options, and then a student or, or none of these, right? Um, and so, so for the purpose of this video right here, we are not going to take every single one of these options because this involves a lot more data cleaning. Let me give you an example: This says software engineer; this also says software engineer, and with AI, these two would typically be combined or standardized to software engineer, but it's not very easy to do that in Power BI. We could do that in Excel, but not really in Power BI or even SQL if we pull this from a SQL database, um, and you can find lots of different, you know, options of that. We have data manager and data manager; if we separated these out, these would be different options when we created our visualizations, and we don't want that. So what we are going to do, uh, and this is going to be kind of a, an easy way out to just make sure that this is pretty clean and doesn't, we don't have a thousand different options, we're going to create this to other. So we're going to simplify this a lot, and then we're going to to use this. So we'll have maybe six or seven options instead of the, you know, let's say 50 that we would have if we actually did the harder work, which is break it out, standardize it, and clean it up that way. So what we're going to do is we're going to click on this right here, and we're going to go up here to split column. In this ribbon up top, we'll go to split column, and we want to do it by a delimiter. And if you notice, let me see if I can move this over; if you notice, we have other, and then we have this parenthesis, and in no other option or way is our parenthesis. So what we're going to do is we're going to use a custom, and we're use this open parenthesis. What that's going to do is it's going to separate it by this parenthesis; it's going to leave the other; it's going to create separate columns, um, just one separate column for each of these, and we can do that at each occurrence or we can do the leftmost, and we really, we only need it for the leftmost because there's only one of these, uh, left-handed or left-sided, uh, brackets or, or what is it, whatever this is called. And then let's go and click okay, and it should create another column. So it's going to have 0.12, and now we have, if we click on this, now we only have these options: We have analyst, architect, engineer, data scientist, database developer, other, and student looking or none. That is what we want; it makes it so much simpler, and it's not perfect, but again, I'm trying to show you what we are able to do in Power BI. So now we're just to remove that column, and we're going to go and do the exact same thing to this one as well. CU I know that we want to use this, and I really wanted to use this one as well, but if we look at this one also, um, there's a lot. So I said, what is your favorite programming language? And people, there were pre-selected answers like JavaScript, Java, C++, Python, R, things like that, and then there was other option, and in this other option, I mean, it was free text, so they can fill it in as they want. I mean, there's four, five, six different ways that people put SQL. That is something I would standardize, and, you know, that would be the way I cleaned it, but that's not how we did it in here. So we're going to do the same thing; we're going to keep that other. So we're going to split this column again; we're use a delimiter, and for this delimiter, though, we're going to use a colon. So we're going to say we're going to do a colon right there; we'll just do the leftmost; we'll click okay, and then we have our options, and it's much simpler now. I really would have rather kept all these and because SQL's in there quite a bit, but, you know, a lot of people don't think SQL is even a programming

Language. So, uh, we're going to delete that column. Now, one that I just skipped and I kind of wanted to go back to is this current yearly salary. I really want to use this. Let's see if we can use it. I here's what I want to do with it, and this is not perfect, um, but for this video, I want to try it. What I want to do is break up these numbers, 6,125, and then take the average of those numbers. Then we'll use some docks in there. So we'll take 106, 125, create that into two separate columns. Then we'll create a third column that will give us the average of those two numbers. So we'll do 106 plus 125 divided by two, and then we'll have the average of that. Now that is not perfect, but it's going to give us at least, you know, an average, a kind of roundabout number because they gave us this range. They said my salary is between 106, 25,000. So if we say that their salary was 112,000, at least gives us it makes it usable. It's a numeric value instead of being this, which is text, which we really we could use and and I'll show you how to do that because we're going to keep this column. I'll create a copy of this, and I'll show you the difference between this and using the average. But for but for this data cleaning portion, let's just try it. Let's see what we can do and see if we can make it work.

So first, let's create a dup duplicate. So we're going to uh duplicate the column. So now we have this copy at the very very end, and we can use this one instead of having to use the original way way way back here. So we're going to leave that one how it is, and we're going to use this one. So let's go ahead and split this one up. We're going to click on the column header, then we're going to click on split column, and we'll do it by digit to non-digit. And if you look at it right here, it's broken it out kind of um in the fact that now in this one we just have numeric values, and in this one we have k-h numeric or just Dash numeric. And now this can be easily cleaned whereas this one we can just completely get rid of because it's only K. So we'll just remove that column. And then in this one, we're going to right-click, we're going to click on replace values, and so if it just has we're just doing a k, we'll replace with nothing. Do okay. And then for the last one, we'll go to replace values, and we'll do the dash or the minus sign, and we'll place that with nothing. And so now we have our values as well. Oh, we also have a plus. Let me get rid of because that's when some people had 250 or 225,000 plus. So for that one, the average is just going to be 225. We'll have to specify that in our Dex. I forgot. But actually if somebody has 220, let me find this plus really quick, uh, let me filter by it because that's a lot faster. What we actually want to do for the purpose of this one is we want to put 225 here so that when we do 225 plus 225 divide by two, it comes out to 225. That's just what we're going to put it as, and there's only two people. So uh I'm actually going to replace this. I'm going to do replace values. I'm going to say plus with 225, and we'll click okay. Awesome. We can unfilter these, select all. So we're going to go right up here to add column, we're going to say custom column, and we're going to go right over here. Actually, let's make it uh average salary. Let's make it average salary. So we're going to insert this, we going to say parentheses, and we're going to say plus this and insert and close the parenthesis divided by two, and it says no syntax errors have been detected. Let's click on okay, and it's giving us an error. So it's saying we cannot apply operator plus to types text and text, which makes perfect sense. These aren't uh numbers. So let's make it a whole number, and let's make it a whole number, and then let's see if this will actually work now or maybe we just need to try a whole another one. So let's try transform or add column custom column. Let's try this all again, see if uh I can make it work. Insert this one plus this one, and we'll do divided by two, and let's try this one, and there we go. So now let's get rid of this column columns, and we can actually remove these ones as well because now we have this um average salary column which when we look at this or when we use this uh we can let me see if I can just move this way way way over. All right, I might cut because this is taking forever. So if you take the average of these two numbers, you'll get 53. If you take the average of 0 and 40, you'll get 20. So now we have this average salary, and again when we get to the actual visualization part, I'll show you why this isn't as useful as having this average salary and just a reminder this is not perfect. Uh, I wouldn't typically do this, especially if I had it in Excel or if I was, you know, creating this survey in a different way. I would probably have a very specific value where they can do it on a slider, but this is how it is. So we've at least made it usable or more usable in my mind, and we have a few other things that we can change like what industry do you work in where we can break this one out. So I'm going to go ahead and break this one out as well as this one right here, which country do you live in. I'm going to breako both of those out to where it's the country or other. I'm not going to have these other values, although there are a lot of them because there's a lot of people who live in these different countries, but we can't really do that super well in here because again the same issue kept happening. Argentina, Argentina, Argentine, a Australia. So we can't normalize those values unless we spend just copious amount of time doing that. So I'm going to go ahead and do these. I'm going to fast I'm going to fast speed this so it goes a lot faster. So I'm just going to go silent and let this happen really quick, and then we'll get to the end and we'll actually start building our visualizations.

All right, so we've split them up, and as you can see, we have all these options as well as other, and I think, you know, there is let me tell you there is so much more that we could do with this. I mean, just so many other things, but this is like what the bare minimum of what we need for this project. So let's go ahead and close and apply this, and if we need to come back at any point and actually fix anything or change anything, we can. So it's not like that's permanent. Um, so as you can see, we have everything over here. We have all of our data as it is transformed in here as well, and and now we can start building out our visualization. Let's go back to our report, and let's start building something out. All right, so let's add a title to our dashboard. Make this right at the top. Call this the data professional survey breakdown, and let's make that quite a bit larger. Make it bold, why not, and we'll put that in the center. And now let's um let's add some effects. Let's change that background to something like it's too dark. Something like this, and I do not like that bold. Let's take that off. There we go. So something like this just has a quick title to what we're about to do, what we are about to build. So we're going to start off with the most simple visualizations that we're going to do, and we'll kind of work our way towards kind of the harder ones. So the first one that we're going to start off with is a card, and the cards are obviously like just super super easy. They usually just display one piece of information. So we're going to go right over here to the very bottom at the unique ID, and we're going to select it, and we're going to say a count of distinct or count. It doesn't matter. Um, it says 630 count of unique ID. Now we're not going to keep that as is. We're actually going to go right over here. We going to say rename for this Visual, and it says count of unique ID, but we're going to say count of survey takers, and you can say whatever you want here, but in in general that is what it is. We're we're counting how many people um, you know, took this survey, and that's just a kind of a total. Maybe I should say total amount or of survey takers, but you can say count of survey takers, how many people took the survey. So let's click out of there. Let's click on card. Let's make it about the same size. We're going to drag it up here and try to make them about the same. We will in a little bit. We'll make them the same size, um, but for this one, we're going to look at age. So we're going to look at current age. So I'm going to click on that, and we'll say the average age. So our average age taker is almost 30 years old. So let's go right over here. We're going to say rename for this visual. We'll say average age of survey o this might be too long. Average age of survey taker. Again, name it whatever you'd like. So again, these are meant to be high-level numbers. So when somebody's looking at your dashboard, they can just really quickly glance at this and know exactly what it is instead of like some of these other visualizations that we're about to create. They don't really have to dig into it, look at the x-axis, the y-axis, the the different uh Legend colors and whatnot. They can just see these high numbers and get a really quick glance of the data.

Now let's create our first visualization, and what we're going to do for that one is a clustered bar chart. So let's go ahead and click on the clustered bar chart. We can create as small or as large as we'd like, and for this one, we're going to be looking at the job titles. Now remember we kind of change the job titles or, you know, U transform those if you want to say that. So we're going to look at Job titles, and then we're going to look at their average salary. And if you remember, we transformed that one as well. We have all average salary. Now this one is it looks like a text right now, so it may not work properly, and what we're actually going to do is go over here. I want to see the average salary. So let's click on average salary and see if we can change this data type from a text to a decimal number. Let's click yes. I forgot to do that when we were transforming it, and there we go. This is perfect. Um, so now we can go back, and we can select our average salary, and as you can see, it has this um this function symbol. So now we can click on it, and it'll look a lot better. And although this says average salary as the title, it's actually doing a count or the sum. So we can click average right here, and what we want to do is actually break this down by the job title. And so now we can see data scientists are making the most by far. They're making an average of 93,000, at least from the survey takers that took it. Then we have our data Engineers making 65,000. Data Architects are making 63, and then where the data analysts data analysts are right here making 55. So again, we had 630 people take this survey, and so the vast majority of them were data analysts. So this one's probably the most accurate out of all of them. And I actually don't like how this looks as the clustered bar chart. Let's try the Stacked bar chart. Put this as the legend. That's more what I was going for. I don't know. I didn't want as skinny because when you're doing this one, it typically they have multiple options per um uh x-axis, and so I think that's why it was that little skinny line, but this one is more what I was looking for. But let's make that smaller, and let's definitely change that title because good night. Um, this is like incredibly long. Let's go over here to this format visual. We'll go to the general, the title, and and we're just going to say average salary by job title, just like that. And this looks a lot better. Now we're not going to kind of format all our whole dashboard yet. We're going to create our visualizations, and then we're going to kind of organize everything and kind of play Tetris with it to make it look the best. So we're just going to minimize this and put it right up here for now, um, but we will go back back and kind of make everything look better at the end. And actually while we're here, I also want to change this as well. So rename for this, we're going to say job title. Oops, why did I do that? Job title, and for this one, we're just going to say name average salary. There we go. Looks much better, much cleaner. Uh, took away a lot of the anxiety that I was feeling about 2 minutes ago when we first put that up there. So let's go on to our second visualization.

The next one that I'm interested in is actually what programming language people were using the most. So we have salary. There's a thousand different things we can look at in here, but I want to know, you know, what is people's favorite programming language. So let's take a look at that. So we have favorite programming language. Let find that. So we have our favorite programming language, and we also have how many people actually took are the unique people. So right now this is columns. We don't want that. Let's um let's do a clustered column chart. Click on this right here, and it looks like here we go. That is kind of what we're looking for. And instead of count of unique ID, we'll say count of let's do count of Voters, and for favorite programming language, we'll say favorite oop favorite programming language, and get rid of that as well. And then we're going to go into here also and change the title and say favorite programming languages or favorite pro programming language, just like this. Now let's make this a lot bigger so you can see it, but really quickly at a glance, you can see Python is by far the most popular. Are other C++, JavaScript, Java. Now all we're seeing is the count, so it's all the same. It's just blue. We can see how many people voted for each one, but if we wanted to break it out similar to how we did with the job titles, we could still do that. So all we'd have to do is break it out, uh, bring this job title down to the legend, and now breaks out like this, and that's not exactly what I was going for. I was going more for something like this where we can see the still the whole count, but now we can see who is actually voting for these things. So I'm just not a huge fan of the colors that are pre-selected here and kind of the whole theme of this dashboard. At the very end, we're going to completely revamp this, change a bunch of colors, the background, and make this look a lot nicer rather than just the white background like we have it. Um, and so for now, let's just make this a lot smaller and put it into this corner. These will not be staying there, but we need to we need room to create our next visualizations and just a cleaner space to do things.

Now the next thing that I really want to include is a way to break down where they're from, their country, because especially something like salary is very dependent on your country, whereas the average salary in the United States for a data analyst may be like 60,000 in another country. It could be 20,000. That could bring down the average quite a bit. So we need a way to be able to break that down. Now we can do something like a field map, and there's no problem with that at all, um, but you know, for what we're building, what we're creating, it's not probably going to work out the best. I mean, this looks okay. We could stick it in the corner or something, um, and you can do that, and that's perfectly fine. I think what I'm going to do is something like a tree map, which I don't use a lot, but I want something where they can just click on it, they can look at the values distinct, they can look at the values and just click on it, and it'll be right there for them. So they don't have to filter it out on their own or no geography, and look at this map. They can just re Canada, other, United Kingdom, India, United States, and click on that. And so, for example, let's click over here on United States. The numbers change quite a bit. Now the average salary for a data scientist is 139,000. For data analyst, it's 80. And if we look at India, you know, the average salary for a data scientist is 68. The average salary is 26 for a data analyst. That doesn't mean that they make less money in India. That just means that the cost of living is probably lower in India. Therefore, they don't need the higher US dollar salary because again, this was all done in US dollars. So just something to think about. Uh, let's click out of that. So we'll keep that one as well. So now let's create our next visualization, and this is one that I do not get to use enough in my actual job. So we're going to use it in this project, um, and it's going to be this gauge right here. So let's add that one. Put it right over here. We're going to add two of those. Let's just go ahead and add another one while we're at it. We're going to have them kind of like right here, right next to each other. The first one, and these ones are really good for kind of looking at these kind of surveys, and I don't get to work with surveys enough, but we can see, you know, how happy are they in terms of work-life balance. So we can add that. We're going to add work-life balance, um, and right now it's doing a count, and if we don't have minimum or maximum values in there yet, so it's going to look kind of weird, but we're going to look at the average rate or the the average score of these. Then we're going to pull this over to the minimum value, and we want to put that at the minimum, and pull this over and add the maximum value. So now it actually has 0 to 10, and it shows that the average person is happy with which one was this? Their average person is happy with their work-life balance. Uh, they rate about a 5.74 overall. Now let's really quickly change the title of this because this is ridiculous. I want to say happy with work-life balance. So this is their rating, uh, you know, change it to whatever title you want. That's what I'm going to do, and we'll also do happy with their salary. Let's click on salary. We'll add that to minimum, and we'll add the maximum value as well to make sure that we know how to use that, and then we'll take the average. So not many people are happy with their salary. I'm just finding out. I mean, this is a real survey. This is real data. So I mean, it's uh pretty interesting. Let's go to the title. Let's go to happy with or maybe it's happiness. Happiness with salary. Maybe that's what we should make it, and I'm going to change that over here as well. I think it sounds better. Some of this I've already planned out. Some I haven't. This is not something I've planned out. So uh so we're going to say happiness with work-life balance, happiness with salary. Really interesting. Um, we may go back and tweak these just a little bit in the future, but the very last visualization that we're going to do is mail versus female. Kind of got to have that in there. Um, I don't typically like pie charts and donut charts, but uh, you know, I'm feeling I'm just feeling it. So let's try it, um, and we will do see. Let's make this larger, so we have male female, and what do we want to look at? Like what do we want to measure? So we have male versus female. We can measure anything.

Um, but maybe what we'll do is the average salary again. I mean, we've kind of only looked at salary once in this one, right here. Um, and a little bit of like how happy they are, but we'll look at the average salary between males and females, and then we'll look at not the current age—Oops, I meant average salary—and then we'll look at the average, and it looks like the average salary is actually really close versus males versus females: 55 for female versus 53 for male. So actually, the females are a little bit higher—congratulations—so they're just a little bit higher in terms of pay.

So now we need to start organizing all of this, cleaning it up, making it look a lot better than it does right now. It looks great, uh, you know, but we can do a lot more with this. So I'm gonna—I'm—we're—we're going to keep these or all these kind of over on this left-hand side. I'm gonna put this—I want this up here. We also need to change that title. I want this up here. Um, and again, we're going to kind of change the theme as we go. I just want to format it right. I'll have it just like this. Let's change the title of this. Let's title, and we're going to say "Country of Survey Takers." Uh, I'm not—the—the survey takers—I'm not really stuck on that. If you find something better, you think of something better, I would go with that, but um, you know, it definitely doesn't look bad. And where did this—where my other visualization go? There it goes. Um, I think this one I want to make kind of more tall, um, so I might move it this way. Jeez, this is such a—I hate—I hate having a lot of visualizations on here; it just really is annoying to me. So what we're going to do, I think we're gonna step this to the side, put this to the side as well, make it to where it's just okay. I didn't want it to cut off; we'll do that. Might make these—make these a little bigger actually. So I want it to kind of match the size like right there. I'll match this perfect. This one I kind of want to bring over here and bring it down a little bit, maybe something like this, maybe. I'm not sure. I'm not—I'm not sold on that.

Um, I added a few different visualizations that I didn't have in my original, so now I'm kind of having to do this on the fly. So um, I might fast forward some of the parts where I'm like really thinking about it or taking too much time on it, but I'm going to bring this down a little bit actually, because I don't like how close that is to um the—the text above it. But one thing we do need to do, I'm going to put this up kind of like this. I think that looks fine. I think I'm going to put this at the very bottom. So let's make some room for it. All right, just like that, stretch it to the side, and we'll lower it, and I think we'll keep that as is, kind of like this. Um, okay, there's a lot going on in here, and there are some things I'm just noticing as we're walking through this that I kind of missed, um, like I need to change some titles and stuff like that. So let me go ahead and change some of those things. So we're going to do title: do "Average Salary by Gender" or "by Sex," do like that: "Average Salary by Sex." I also don't like that it's in the middle. Um, I don't like that it's on the outside; I want them on the inside for this. So let's go to the details, let's go to inside, and see if that looks any better. Oh, that looks terrible. Um, let me see if I can change that. Maybe I don't—no, I definitely want it—I guess we'll do outside. I—you can't even see the information. Oh, the decimal is crazy long. Um, let me go and see if I can change that decimal to just like a whole number or like 1.1, uh, because that's a problem. So maybe I need to go over here to the value. All right, so I think I want to change this one; it's just not working out exactly how I wanted. And you guys know if I make mistakes, I'm going to keep it in here so you guys can see it. I—I hoped that this was going to turn out better, but it didn't.

Um, one that I do want to add, because this is kind of a—a breakdown and a nice visualization, I want to add this difficulty piece. So I want to add this: "How difficult was it for you to break into data science?" Let's get rid of these, and I want to click on this really quickly, see what it gives us—values. Okay, so now this shows us percentages um of how easy it was. Again, it's neither easy nor difficult, difficult, easy, very difficult, very easy. These numbers make abs—no sense. We need to kind of order them a little better. So I'm going to come over here to slices. We have our colors over here. We want very difficult to be like the most difficult, um, so we're going to make that red, and then we want difficult to be maybe like an orange. Let's see if we can find an orange. There we have an orange. This does not look red enough. There we go. Oh, no, no, no. Very difficult is red, difficult is orange. We have neither easy nor difficult, and that's kind of a neutral—let's see if we have something neutral in here, kind of like this yellow. I don't know, let's try it out. Then we have easy and very easy, and these will be like our blues. So I'm going to keep that—I'm going to keep that kind of like a dark blueish, and then our blue for super easy is just going to be like really blue. U, and that doesn't look bad. The—I mean, look, I'm—I'm not a color person; I—I'm not great with colors. And we're going to kind of organize this in just a little bit, but this looks better to me. Um, but we need to change up some stuff as well, like the title. Need to do "Difficulty to Break into Data." There we go. And we're also going to change this title right here. Here, we're just gonna say "Difficulty." Difficulty. Difficulty. This looks better to me. Um, again, not perfect, and there's a thousand different things you could have done, but that's just what we're going to do.

I need to go through here and see what I need to change. So right off the bat, I can see I need to change this—um, to—let's see, right here, going to rename this job title just like we did in this one right here—count of voters, that's fine, programming language, breaking into, difficulty, happiness, happiness, average, count. Okay, okay. So what we have here is very close to a finished product. Now, it's not 100% complete. I mean, I—I do want to make it look a little nicer rather than just the typical white. So what we're going to do, we're going to go up here, we'll go to—what is it?—View, and we have all these different fil—filters, and we're just going to play around with it, see if we can find something that we like. Um, this doesn't look too bad. It's—uh—not really my style. U, we can do this one, Frontier. This is pretty neat. I kind of am digging this. We might come back to it. I like the natural tones. I don't know why I said tones like that, but I did. Um, this one's not bad, but I don't—I don't—it's not—that's not my—I don't like how dark that is. Um, and so maybe it's like, you know, uh, we change like the background color of all of these as well as match it with—match it with something else, whatever you want. Genuinely, you customize this however you want. I kind of like this one; it's kind of groovy, man. And um, it's not perfect by any means, but what we can do, and we can customize this current theme. We can come in here, customize this theme however we'd like. I personally don't want color five, which is the data analyst color. I don't like it. I want to go out—go and change it because I don't like it, but I don't really like that color per se, you know. I might want to choose a different color, um, but it has to be like this muted, like that; it has a style to it. So you can come in here, and you can customize this and make it however you'd like, and—and really mess around with it, play around with it. For me, uh, I'm just going to keep it how it is because I don't really want to mess with it, break it, or anything like that. So um, let me just—that up just a tiny bit.

So this is it. This is the project. I hope that it was helpful. Um, I am not joking when I say that I'm—because I'm gonna do a different project. I'm gonna go really in depth in another project. It's probably going to be like a two-hour project; it's gonna be crazy long, um, well, for a YouTube video, but I can see doing a thousand different things with this data, creating a really great dashboard, really cleaning the data, which is a large part of—of actually doing this, and we didn't do much data cleaning at all. There's just so much you can do with this, and so really dig into this, see what you like, see what you don't like, see what you want to clean, what you don't want to clean. You could put it in SQL, you could put it in um Excel, and just—and just standardize the data to make it a lot more usable. Do whatever you want with it. I mean, I—I took this survey for you guys that we could use it, so go out and use it and make the best dashboard that you can possibly do. So I hope that this was helpful. I hope that you enjoyed this. Thank you so much for watching this video. If you like this—thank you so much for watching. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

What's going on, everybody? Welcome back to another video. Today we're going to be starting our Python tutorial [Music] series. Now, I am extremely excited for this series. We're going to be walking through all the things that you need to know to get started in Python. We'll be looking at variables, data types, for loops, while loops, operators, and a ton more. After this beginner series, we're going to be going into another set of series where we look at pandas, matplotlib, seaborn, web scraping, and more. Now, in this video, we're just going to be setting up our environment to where we can learn Python in future videos. In this series, we're going to be using Jupyter Notebooks for all of our tutorials because I feel like it's a really great place to learn the basics, but then in future videos I'll show you different IDEs that you can use for your Python code. I genuinely cannot wait to get started on this series. I absolutely love Python. So without further ado, let's jump on my screen. I'm going to show you how to install Jupyter Notebooks.

All right, so let's get started by downloading Anaconda. Anaconda is an open-source distribution of Python and R products. So within Anaconda is our Jupyter Notebooks as well as a lot of other things, but we're going to be using it for our Jupyter Notebooks. So let's go right down here, and if I hit download, it's going to download for me because I'm on Windows, but if you want additional installers if you're running on Mac or Linux, then you can get those all right here. Now, if you are running on Windows, just make sure to check your system to see if it's a 32-bit or a 64-bit. You can go into your about and your system settings to find that information. I'm going to click on this 64-bit; it's going to pop up on my screen right here, and I'm going to click save. Now, now it's going to start downloading it. It says it could take a little while, but honestly, it's going to take probably about 2 to 3 minutes, and then we'll get going. Now that it's done, I'm just going to click on it, and it's going to pull up this window right here. We are just going to click next because we want to install it. This is our license agreement; you can read through this if you would like. I will not; I'm just going to click I agree. Now we can select our installation type, and you can either select it for just me, or if you have multiple admin or users on one laptop, you can do that as well. For me, it's just me, so I'm going to use this one as it recommends. Now it's going to show you where it's installing it on your computer. This is the actual file path. It's going to take about 3.5 gigs of space. I have plenty of space, but make sure you have enough space, and then once you do, you can come right over here to next. And now we can do some advanced options. We can add Anaconda3 to my path environment variable, and when you're using Python, you typically have a default path with whatever Python IDE or notebook that you're using. I use a lot of Visual Studio Code, so if I do this, I'm worried it might mess something up, so I am not going to do this. It also says it doesn't recommend it. Again, messing with these paths is kind of something that you might want to do once you know more about Python, so I don't really recommend you having this checked. We can also register Anaconda3 as my default Python 3.9. You can do this one, and I'm going to keep it this way just so I have the exact same settings as you do. So let's go ahead and click install, and now it is going to actually install this on your computer. Now once that's complete, we can hit next, and now we're going to hit next again, and finally we're going to hit finish. But if you want to, you can have this tutorial and this getting started with Anaconda. I don't want either of them, cuz I don't need them, but if you would like to have those, keep those checked, and you can get those. Let's click finish. Now let's go down, and we're going to search for Anaconda, and it'll say Anaconda Navigator, and we're going to click on that, and it should open up for us. So this is what you should be seeing on your screen. This is the Anaconda Navigator, and this is where that distribution of Python and R is going to be. So we have a lot of different options in here, and some of them may look familiar. We have things like Visual Studio Code, Spyder, RStudio, and then right up here we have our Jupyter Notebooks, and this is what we're going to be using throughout our tutorials. So let's go ahead and click on launch, and this is what kind of pop up on your screen. Now I've been using this a lot, um, so I have a ton of notebooks and files in here, but if you are just now seeing this, it might be completely blank or just have some, you know, default folders in here, but this is where we're going to open up a new Jupyter Notebook where we can write code and all the things that we're going to be learning in future tutorials. And you can use this area to save things and create folders and organize everything. If you already have some notebooks from previous projects or something, you can upload them here, but what we're going to do is go right to this new—we're going to click on the drop down, and we're going to open up a Python 3 kernel. And so we're going to open this up right here. Now, right here is where we're going to be spending 99% of our time in future videos. This is where we're going to write all of our code. So right here is a cell, and this is where we can type things. So I can say print, I can do the famous hello world, and then I'll run that by clicking Shift+Enter, and this is where all of our code is going to go. These are called cells, so each one of these are a cell, and we have a ton of stuff up here, and I'm going to get to that in just a second. One thing I wanted to show you is that you don't only have to write code here; you can also do something called Markdown. And so Markdown is its own kind of—you could say—language, but um, it's just a different way of writing, especially within a notebook. So all we're going to do is do this little hashtag—and actually, I think it's a pound sign, but I'm going to call it hashtag—we're going to do that; we're going to say "First Notebook," and then if I run that, we have our "First Notebook," and we can make little comments and little notes like that that don't actually run any code; they just kind of organize things for us, and I'm going to do that in a lot of our future videos, so just wanted to show you how to do that.

Now let's look right up here. A lot of these things are pretty important. Uh, one of the first things that's really important is actually saving this. So let's say we wanted to change the title to—I'm going to do AAA because I want it to be at the beginning—um, so I can show you this. Going to do "AAA New Notebook," and I'm going to rename it, and then I'm going to say save that. So if I go right back over here, you can see "AAA New Notebook." That green means that it's currently running, and when I say running, I mean right up here. And if we wanted to, we go ahead and shut that down, which means it wouldn't run the code anymore, and then we'd have to run up a new cluster. Uh, so let's go ahead and do that. I didn't plan on doing that, but let's do it. So we have no notebooks running, and right here it says we have a dead kernel. So this was our Python 3 kernel, and now since I stopped it, it's no longer processing anything. So let's go ahead and say "Try Restarting Now," and it says "Kernel is ready," so it's back up and running, and we're good to go. The next thing is this button right here. Now, this is an "Insert Cell Below." So if I have a lot of code, I know I'm going to be writing, I can click a lot of that, and I often do that because I just don't like having to do that all the time, so I make a bunch of cells just so I can use them. You can also delete cells. So say we have some code here; we'll say "Here," and we have—"Have Code Here," and then we have this empty cell right here; we can just get rid of that by doing this: "Cut Selected Cells." We can also copy selected cells. So if I hit "Copy Selected Cells," and I can go right here and say "Paste Selected Cells," and as you can see, it pasted that exact same cell. You can also move this up and down. So I can actually take this one and say I wanted it in this location. I can take this cell and move it up, or I can move it down, and that's just an easy way to kind of organize it instead of having to like copy this and moving it right down here and pasting it; you can just take this cell and move it up, which is really nice. Now, earlier when I ran this code right here, I hit Shift+Enter. You can also run, and it'll run the cell below. So you can hit run, and it works properly. If you're running a script and it's taking forever and it's not working properly—at least it's—you don't think it's working properly—you can stop that by doing this: "Interrupt the Kernel" right here. And anything you're trying to do within this kernel, if it's just not working properly, it'll stop it. You can restart it, then you can try fixing your code. You can also hit this button if you want to restart your kernel, and this button if you want to restart the kernel and then rerun the entire notebook. As we talked about just a second ago, we have our code and our Markdown code. We're not going to talk about either of these because we're not going to use that throughout the entire series. The next thing I want to show you is right up here. If you open this file, we can create a new notebook, we can open an existing notebook, we can copy it, save it, rename it, all that good stuff.

Stuff we can also edit it. So a lot of these things that we were talking about, you can cut the cells and copy the cells using these shortcuts if you would like to. We also go to View, and you can toggle a lot of these things if you would like to, which just means it'll show it or not show it depending on what you want. So if we toggle this toolbar, it'll take away the toolbar for us; or if we go back and we toggle the toolbar, we can bring it back. We can also insert a few different things, like inserting a cell above or a sell below. So instead of saying "This plus button," you can just say "A" or "B," we adding above or below. We also have the cell in which we can run our cells, or run all of them, or all above, or all below. And then we have our kernels right here, which we were talking about earlier, where we can interrupt it and restart those. There are widgets; we're not going to be looking at any widgets in this series, but if it's something you're interested in, you can definitely do that. Then we have Help. So if you are looking for some help on any of these things, especially some of these references which are really nice, you can use those, and you can also edit your own keyboard shortcuts.

And now that we walked through all of that, you now have Anaconda and Jupyter Notebooks installed on your computer. In future videos, this is where we're going to be writing all of our Python code, so be sure to check those out so we can learn Python together. Thank you guys so much for watching. I hope you were able to get everything installed correctly. I am super excited for this series ahead of us. If you like this video, be sure to like and subscribe below, and I will see you in the next [Music] video.

Hello everybody. Today we're going to be learning about variables in Python. A variable is basically just a container for storing data values. So you'll take a value like a number or a string; you can assign it to a variable, and then the variable will carry and contain whatever you put into it. So for example, let's go right over here. We're going to say x, and this is going to be our variable. We're going to say is equal to; now we can assign the value to it. So let's say I want to put 22. X is now equal to 22. So we won't have to write out the number 22 in later scripts that we write; we can just say x because x is equal to 22; it now contains that number. So now we can hit Enter and say print; we do an open parenthesis and we'll say x. Now I'm going to hit Shift+Enter, and now it prints out that 22 because we are printing x, and x is equal to 22. This is our value, and this is our variable.

One really great thing about variables is that it assigns its own data type; it's going to automatically do this. So we didn't have to go and tell x that it's an integer; it just automatically knew that 22 is a number. So we can check that by saying type and then open parenthesis and writing x, and we'll do Shift+Enter again, and this says that x is an integer type. Now we only assigned an integer to x. Let's try assigning a string value, or some text, to a variable. So we'll say y is equal to, uh, let's say "mint chocolate chip." I'm feeling some ice cream today, so we'll say "mint chocolate chip." Now if we print that again, we'll do print, open parenthesis, y, and do Shift+Enter; it'll print "mint chocolate chip," and if we look at the type, we can see that the type is a string this time and not an integer. Now again, we did not tell it that x was an integer and y was a string; it just automatically knew this. Let's go, go up here really quickly; we're going to add several rows in here because we're about to write a lot of different variables and really learn in depth how to use variables.

The next thing to know about variables is that you can overwrite previous variables. Right now we have "mint chocolate chip," and that is assigned to the variable y. So if I go down here, I say print y, I hit Shift+Enter, it's going to print out "mint chocolate chip," but if I go right above it, I say y is equal to, and let's say "chocolate," if I print that out, it's now going to say "chocolate," whereas up here, I'm reassigning it to y; it's still going to say "mint chocolate chip." So if I come right down here and I copy this and I'm going to paste this right here, initially it is going to assign y to "chocolate," but then right here it will automatically overwrite y as "mint chocolate chip," and when we hit Shift+Enter, it's going to show "mint chocolate chip." Variables are also case-sensitive. So if I come up here and I say a capital Y—this is a lowercase y, and this is a capital Y—it is going to print out the correct one instead of "mint chocolate chip," and then if I go down here to the print and I type the capital Y, it will give us the "mint chocolate chip."

Up till now we've only assigned one value to one variable, but we can actually assign multiple values to multiple variables. So let's do x, comma, y, comma, z. Z is equal to, and now we can assign multiple values to all of those. So we can say "chocolate," and then we'll do a comma, oops, a comma, then we can say "vanilla," and then we'll do another comma, and we'll say "rocky road." Now this is going to assign "chocolate" to x, "vanilla" to y, and "rocky road" to z. So what we can do is we'll say print, and we'll go print, print, print, and we'll say x, y, and z. So it prints out "chocolate," "vanilla," and "rocky road," and these are our three different values. We can also assign multiple variables to one value, and we can do this by saying x is equal to y is equal to z is equal to, and we can put whatever we would like. Let's do "root beer float." Then we'll come back up here; here we'll copy this, and let's print off our x, our y, and z, and they are all the exact same now.

So far we've really only looked at integers and strings, but you can assign things like lists, dictionaries, tuples, and sets all to variables as well. So let's go right down here. So let's create our very first list. I'm going to say ice_cream is equal to, and that is our variable right there; the ice_cream is our variable. So now we're going to do an open bracket like like this, and we're going to come up here and copy all of these values, and we're going to stick it within our list. So now within ice_cream we have three string values, "chocolate," "vanilla," and "rocky road," all within this list. So what we can do is we can say x, comma, y, comma, z is equal to ice_cream. So now these three values, "chocolate," "vanilla," and "rocky road," will be assigned to these three variables, x, y, and z, and we can copy this print up here, and we'll hit Shift+Enter, and now the x, y, and z all were assigned these values of "chocolate," "vanilla," and "rocky road."

Now something that we just did, which is really important, or something that you really need to consider, is how you name your variables. So right here we have ice_cream. Now this, to me, is exactly how I usually write my variables, but there are many different ways that you can write your variables. So let's, let's take a look at that really quickly, and let's add just a few more, cuz I have a feeling we're going to go a little bit longer than what we have. So there are a few best practices for naming variables. First, I'm going to show you kind of what a lot of people will do; I'll show you some good practices, and I'm going to show you some bad practices as well that you should avoid doing. The first thing that we're going to look at is something called camel case. And let's say we want to name it testVariableCase, oops, case. Now if we have a testVariableCase, the camel case is going to look like this: we'll have lowercase test, and then we'll have uppercase Variable and uppercase Case is equal to; this is what this variable is going to look like, and we can assign it "vanilla swirl," and this is what your camel case will look like. It's going to be lowercase, and then all the rest of those, uh, compound words, or however you want to say that, these letters are going to be capitalized to kind of separate where the words end and begin. Let's go right down here; we're going to copy this.

The next one is called Pascal case. So Pascal case is going to look just a little bit different. Instead of the lowercase at test, it's going to be a capital T in test. So TestVariableCase, again, this is a very similar way of writing it, very similar to camel case, but just a capital at the beginning. Now let's look at the last one, and this one is my personal favorite; this one is going to be the snake case. Now this one is quite a bit different in the fact that you don't use any capital letters, and you separate everything using underscore. So we're going to write test_variable_case. Now typically—let me have them all in there—typically these are the best practices; these are what you typically want to do, but probably the best one to use is this snake case right here. What a lot of people say is that it improves readability. If you take a look at either the camel case or the Pascal case, which you will see people do, it's not as easy to distinguish exactly what it says, and the name of a variable is important because you can gain information from it if people name them appropriately. So when I'm naming variables, I usually write it in snake case because I just find it a lot easier to read because each word is broken up by this underscore.

So now let's look at some good variable names. These are all ones that you can use or could use. Let's do something like test_var. So test_var is completely appropriate. We can also do something like test_var, oops, underscore; we could do underscore test_var. You'll see that often as well; well people will start it with an underscore. You can do testVar, capital T, oops, capital T, capital V in testVar, or you could even do something like test_var2. Now adding a number to your variable is not inherently a bad thing; usually it's semi-frowned upon, but there are definitely some use cases where you can use it. But one thing that you cannot do is do something like putting the two at the front. If you put the two at the front, it no longer works; it won't run properly at all. So we're going to take that out; so we can't do that. So I'm going to use this as an example of what you should not do. You also can't use a dash, so something like test-v2, that doesn't work either, and you also can't use something like a space or a comma, or really any kind of symbol like a period or a backslash or equal sign; none of those things will work within your variable.

Now another thing that you can do within your variable is use the plus sign. So let's assign this; we'll say x is equal to, and we'll do a string; we'll say "ice cream is my favorite," and then we'll do a plus sign, and we'll say period. Now what this will do is it will literally add these two strings together. So let's do print, and we'll do x. So now it says "ice cream is my favorite." One thing that we cannot do in a variable is we cannot add a string and a number or an integer. So we can't do "ice cream is my favorite 2." If we try to do that, it will give us this error right here. So in this error, it's saying you can only concatenate a string, not an integer, to a string; so only a string plus a string for this example. You can also do—and we'll say x is equal to—or we'll say y; we'll say y is equal to 3 + 2, and it should output 5 because you can also do an integer and an integer now.

So far we've only been outputting one variable in the print statement, but you can actually add multiple variables within a print statement. So let's go right down here; we're going to say—let's give it some more right there—so we'll say x is equal to "ice cream", and we'll say y is equal to "is", and then the last one, z is equal to "my favorite", and we'll do a period at the end. Now we can go to the bottom and we can say print x + y + z, and when we enter that and when we run and when we run that, we get "ice cream is my favorite." Now we can actually add a space before "is," a space before "my," and when we hit Shift+Enter, it says "ice cream is my favorite." You can also do this exact same thing with numbers as well. So we'll say x equal to 1, 2, and what z is equal to 3, so this should equal 6.

Now one thing that we tried to do was assign to one variable a string plus an integer, and that did not work, but what you can do is you can take something like this and you can say "ice cream," and we'll get rid of this one, and we'll get rid of the z. Now saying plus is actually not going to work. Let's try running this. So again, we can't concatenate these, but what we can do in the print statement is we can separate it by a comma. So when we add this comma, it should work properly. Let's hit Enter, and it says "ice cream 2." Again, this makes no sense, but you are able to combine a string and an integer separating by a comma. Now this is the meat and potatoes of variables. There are some other things as well, but some of those things are a little bit more advanced and not something I wanted to cover in this tutorial, although we may be looking at some of those things in future tutorials, but this is definitely the basics; what you really, really need to know about variables. I hope that this video was helpful. If it was, be sure to like and subscribe below, and I will see you in the next [Music] video.

Hello everybody. Today we're going to be talking about data types in Python. Data types are the classification of the data that you're storing. These classifications tell you what operations can be performed on your data. We're going to be looking at the main data types within Python, including numeric, sequence type, set, Boolean, and dictionary. So let's get started actually writing some of this out, and first let's look at numeric. There are three different types of numeric data types: we have integers, float, and complex numbers. Let's take a look at integers. An integer is basically just a whole number, whether it's positive or negative. So an integer could be a 12, and we can check that by saying type; we'll do an open parenthesis and a close parenthesis, and if we, if we say the type of 12, it's going to give us an integer; or if we say a negative -2, that is also an integer. We can also perform basic calculations, like -2 + 100, and that'll tell us it is also an integer. So whether it's just a static value or you're performing an operation on it, it's still going to be that data type if those numbers are whole numbers, whether negative or positive.

Now let's take this exact one and let's say 12, and we'll do + 10.25. When we run this, it's no longer going to be a whole number; it'll now be a float. So let's check this. Now this is a float type because it's no longer a whole number; it's now a decimal number. And the last data type within the numeric data type is called complex. Let's copy this right down here. Now personally, this is not one that I've used almost ever, but it is one just worth noting. So you can do 12 plus, and let's say 3j. And if we do this, it's going to give us a complex. The complex data type is used for imaginary numbers. For me, it's not often used, but if you do use it, j is used as that imaginary number. If you use something like c or any other number, it's going to give you an error; j is the only one that will work with it.

Now let's take a look at Boolean values. So we'll say Boolean. The Boolean data type only has two built-in values: either True or False. So let's go right down here and say type True, and when we run this, it'll say bool, which stands for Boolean. We can do the exact same thing with False; that is also Boolean. And this can be used with something like a comparison operator. So let's say 1 is greater than 5, and let's check this. This is giving us a Boolean because it's telling us whether 1 is greater than 5. Let's bring that right down here; this will give us a False. So it's telling us that 1 is not greater than 5. And just as we got a False, we can say 1 is equal to 1, and this should give us a True.

So now let's take a look at our sequence type data types, and that includes strings, lists, and tuples. Let's start off by looking at strings in Python. Strings are arrays of bytes representing Unicode characters. When you're using strings, you put them either in a single quote, a double quote, or a triple quote. I call them apostrophes; it's just what I was raised to call them, but most people who use Python call them quotes. So right here we have a single quote, and that works well. We can do a double quote, and that works also, and as you can see, they are the exact same output. And then we have a triple quote, just like this, and this is called a multi-line. So we can write on multiple lines here. So let's write a nice little poem. So we'll say, "The ice cream vanquished my longing for sweets. Upon this diet I look away; it no longer exists on this day." And then if we run that, it's going to look a little bit weird; it's basically giving us the raw text, which is completely fine, but let's call this a multi-line, and we're going to call this a variable multi_line, and we're going to come down here and say print, and before I run this, I have to make sure that this is run. So now let's print out our multi_line, and now we have our nice little poem right down here.

Now something to know about the single and double quotes is how they're actually used. So if we use a single quote and we say, "I've always wanted to eat a gallon of ice cream," and then we do an apostrophe at the end, obviously something went wrong here. What went wrong is when you use a single quote and then within your text, within your sentence, you have another apostrophe, it's going to give you an error. So what we want to do is whenever we have a quote within it, we need to use a double quote. These double quotes will negate any single quotes that you have within your statement; they won't, however, negate another double quote, so you need to make sure you aren't using double quotes within your sentence if you want to do something like that; you need to use the triple quotes like we did above. So we can do double, double, and then let's paste this within it, and anything you do within these triple quotes will be completely fine as long as you don't, don't do triple quotes within your triple quotes. We'll say "this is wrong." So even though it's between these two triple quotes, it doesn't work exactly. Again, you just have to understand how that works; you have to use the proper apostrophes or quotes within your string. And just to check this, we can always say, here's our multi_line; we can always say type of multi_line, and that is still a string.

One really important thing to know about strings is that they can be indexed. Indexing means that you can search within it, and that index starts at zero. So let's go ahead and create a variable, and we'll just say a is equal to, and let's do the all-popular "Hello, world!" Let's run this, and now when we print the string, we can say a, and we're going to do a bracket, and now we can search throughout our string using the index. So all you have to do is do...

a colon and we can say five. What this is going to do is going to say zero position zero all the way up to five, which should give us the whole "hello". I believe let's run this, and it's giving us the first five positions of this string.

We can also get rid of the colon and just say something like five. And then when we run this, it's actually going to give us position five. So this is 0, 1, 2, 3, 4, and then five is the space. Let's do six so we can see the actual letter, and that is our "w".

We can also use a negative when we're indexing through our string. So we could say -3, and it'll give us the "L" because it's -1, 2, and 3. We can also specify a range if we don't want to use the default of zero. So before we did 0 to 5, and it started at zero because that was our default, but we could also do 2 to 5. Let's run this, and now we go position 0, 1, and then we start at 2: "LL".

Now we can also multiply strings. And we have this: "a = hello world". So we can do a * 3. And if we run this, it'll give us "hello world" three times. And we can also do A + a, and that is "Hello World hello world".

Now let's go down here and take a look at lists. Lists are really fantastic because they store multiple values. The string was stored as one value, multiple characters, but a list can store multiple separate values. So let's create our very first list. We'll say `list` really quickly, and then we'll put a bracket, and a bracket means this is going to be a list. There are other ones like a squiggly bracket and a parenthesis; these denote that they are different types of data types. The bracket is what makes a list a list. So to keep it super simple, we'll say 1, 2, 3, and we'll run this, and now we have a list that has three separate values in it. The comma in our list denotes that they are separate values, and a list is indexed just like a string is indexed. So position zero is this one, position one is the two, and position two is the three.

Now, when we made this list, we didn't have to use any quotes because these are numbers. But if we wanted to create a list and we wanted to add string values, we have to do it with our quotes. So we'll say "cookie dough", then we'll do a comma to separate the value, and then we'll say "strawberry", and then we'll do one more, and this will just be "chocolate". And when we run this, we have all three of these values stored in our list.

Now, one of the best things about lists is you can have any data type within them. They don't just have to be numbers or strings; you can basically put anything you want in there. So let's create a new list and let's say "vanilla", and then we'll do 3, and then we'll add a list within a list and we'll say "Scoops", "spoon", and then we'll get out of that list, and then we'll add another value of `True` for Boolean. And now we can hit Shift+Enter, and we just created a list with several different data types within one list.

Now let's take this one list right here with all of our different ice cream flavors. We'll say `ice_cream =` this list. Now, one thing that's really great about lists is that they are changeable; that means we can change the data in here. We can also add and remove items from the list after we've already created it. So let's go and take `ice_cream`, and we'll say `ice_cream.append`, and this is going to append it to the very end of the list, do an open parenthesis, let's say "salted caramel". Now when we run this and we call it just like this, it's going to take this list, add "salted caramel" to the end, and we'll print it off. And as you can see, it was added to the list. And just like I said before, let me go down here; we can also change things from this list. So let's say `ice_cream`, and then we need to look at the indexed position, so we're going to say `[0]`, and that's going to be this "cookie dough" right here. We can say that is equal to, so we can now change that value. So let's call that "butter pecan". And now when we call it, we can now see that the "cookie dough" was changed to "butter pecan".

Another thing that you saw just a little bit ago is something called a list within a list, basically a nested list. So we had "Scoops", "spoon", `True`. Let's give this, and we'll say `nested_list =`. Now when we run this, we now have this nested list. So if we look at the index and we say `[0]`, we'll get "vanilla". If we say `[2]`, we'll get "Scoops" and "spoons". Now, since we have a list within a list, we can also look at the index of that nested list. So let's now say `[1]` and that should give us just "spoon". And you can go on and on and on with this; you can do lists within lists within lists, and all of them will have indexing that you can call.

Now let's go down here and start taking a look at tuples. So a list and a tuple are actually quite similar, but the biggest difference between a list and a tuple is that a tuple is something called immutable; it means it cannot be modified or changed after it's created. Let's go right up here; we're going to say `tuple`, and let's write our very first tuple. So we'll say `tuple_scoops =`, and then we'll do an open parenthesis. Now these open parentheses you've seen if you do like a `print` statement, but that's different because that's executing a function; this is actually creating a tuple, which is going to store data for us. So we'll say 1, 2, 3, 2, and 1. Let's go ahead and create that tuple, and we can just check the data type really quickly, and it's a tuple. And just like we saw before, a tuple is also indexed. So if we go at the very first position, which is a one, we will get the output of a one. But we can't do something like `.append` and then add a value like three. If we do that, it's going to say "tuple object has no attribute 'append'". It's just because you cannot change or add anything to a tuple, just like we were talking about before. Typically, people will use tuples for when data is never going to change. An example for this might be something like a city name, a country, a location, something that won't change. They definitely have their use cases, but I don't think they're as popular as just using a list.

So now let's scroll down and start taking a look at sets. But really quickly, let me add a few more cells for us, and let's say sets. Now, a set is somewhat similar to a list and a tuple, but they are a little bit different in the fact that they don't have any duplicate elements. Another big difference is that the values within a set cannot be accessed using an index because it doesn't have an index because it's actually unordered. We can still loop through the items in a set with something like a `for` loop, but we can't access it using the bracket and then accessing its index point. So let's go ahead and create our very first set. So we're going to say `daily_ice_cream_pints`, then we're going to say `=`, and to create a set, we're going to use these squiggly brackets. I don't know if there's an actual name for those if I'm being honest; I call them squiggly brackets, and that's what we're going to go with. We're going to put in a 1, a 2, and a 3. So let's go ahead and run this, and let's look at `type`, and as you can see, it is a set. Now when we print this out, it's going to show us 1, 2, and 3, and those are all the values within our set. But if we copy this and we'll say `daily_pints_log`, this is going to be every single day, maybe I had different values. Now when we run this and we do the exact same thing, now when we print this, it's going to have just the unique values within that set.

Now, a use case for sets, and this is something that I've done in the past, is comparing two separate sets. Maybe you have a list or a tuple, and you convert that into a set, and that will narrow it down to its unique values. Then you can compare the unique values of one set to the unique values in another set, and then we can see what's the same and what's different. So let's go down here and let's say `wifes_ice_cream_daily`, and we'll just copy this right here. We'll say `=`, let's do our squiggly lines, let's do 1, 2, let's do just random numbers. So now this is my daily log, and this is my wife's daily log, and now we can compare these values. So let's go right down here; let's say `print`. We'll do `my_daily_logs`, and then we'll do this bar right here, and this is going to show us the combined unique values; it's basically like putting them all in one set and then trimming it down to just the unique values. So we'll take `wifes_daily_pints_log`. When we run this, we actually need to run this first. When we run this, we should see all the unique values between these two sets. And so as you can see, 0, 1, 2, 3, 4, 5, 6, 7, 24, 31. So these are all the unique values between these two sets. We can also do another one, and instead of this bar, we're going to do this symbol right here, which I believe is called an ampersand; don't quote me on that, but when we run this, it's going to show what matches; that means which which ones show up in both sets. So the only ones that show up in both sets are 1, 2, 3, and 5. We can also do the opposite of that by doing a minus sign, and this is going to show us what doesn't match. And so we have 4, 6, and 31. Now, where is our 24? That was in our wife's daily pints log; it's in this one, but we're subtracting the values on this one. So let's reverse this, and we'll say `daily_pints_log`, and let's run it. Now those are our other values. So we're taking the values of this, and then we're subtracting all the ones that are the same and getting the remaining values. And then for our last one, we can get rid of this, and we'll do this symbol right here, and this is going to show if a value is either in one or the other but not in both. So let's run this. So these values are completely unique only to each of those sets.

Now the very last one that we're going to look at in this video is dictionaries. So let's go right down here; let's add a few cells, and let's say dictionaries. Now I saved dictionary for last because this one is probably the most different out of all the previous data types that we've looked at. Within a data type, we have something called a key-value pair; that means when we use a dictionary, it's not like a list where you just have a value, comma value, comma value. We have a key that indicates what that value is attributing it to. So let's write out a dictionary to see how this looks. So we're going to say `dictionary_cream`, and just like a set, we use a squiggly line, but the thing that differentiates it is that in a dictionary we'll have that key-value pair, whereas in a set each value is just separated by a comma. So let's write `name`, and this is our key, and then we do a colon, and this is then where we input our value. So we're going to say "Alex Freeberg", and then we separate that key-value pair by a comma, and now we can do another key-value pair. So we'll say `weekly_intake`, a colon, and we'll say "five pints of ice cream", do a comma, and then we'll do `favorite_ice_creams`, and now what we're going to do is we're going to put in here a list. So within this dictionary, we can also add a list. We'll do "MCC" for mint chocolate chip, and then we'll add "chocolate", another one of my favorites. So now we have our very first dictionary. Let's copy this and run it, and let's just look at the `type`, and as you can see, it says that this is a dictionary. Let's also print it out. Now if we want to, we can take our `dictionary_cream` and say `.values`, with an open parenthesis, and when we execute this, we'll see all of the values within this dictionary. So here's our values of "Alex Freeberg", 5, "mint chocolate chip", and "chocolate". We can also say `.keys`, and when we run this, all of the keys: `name`, `weekly_intake`, and `favorite_ice_creams`. And we can also say `.items`. So this key-value pair is one item, and this key-value pair is another item.

Now, one difference between something like a list and a dictionary is how you call the index, but you can't call it by doing something like this where you just do a bracket, oops, and say `[0]`. So this would, in theory, take this very first one right, our very first key-value pair; that's going to give us an error. How you call a dictionary is actually by the key, so it doesn't technically have an index, but you can specify what you want to call and take it out. So we're going to say `name`, and this is going to call that key right here, and when we run this, we'll get the value, which is "Alex Freeberg". One other thing that you can do is you can also update information in a dictionary, which we can't with some other data types. So for this, for the `name`, it was "Alex Freeberg". Now let's say "Freeberg". And when we update that, I'm also going to print the dictionary, get rid of this. So it's going to update "Christine Freeberg" in that value of the `name`. So let's go ahead and run this, and now it changed the `name` from "Alex Freeberg" to "Christine Freeberg". We can also update all of these values at one time. So let's copy this, and I'm going to put it right down here. I'm going to say `dictionary.cream.update`, then we're going to put a bracket, or not a bracket, but a parenthesis around these. So now what we're going to do is update this entire thing. Let me take this, say `print` this dictionary. Now we can update this to anything we want. So instead of here, I can say I'll say `weight`, and because of all that ice cream, I now weigh 300 lbs. So let's run this, and as you can see, it did not delete our key-value pair right here; instead, it just added to it. When you're using the `.update`, we can't actually delete; that's the `del` statement, and I'll show you that in just a second. But all we did was add this new value; it also is going to check and see if you changed anything with your key-value pair. So we can go in here and change this value, and we'll say 10. So now when we run this, the value of this key-value pair was changed. But let's say we do want to delete it. We'll say `del`, that stands for delete, part of this `dictionary_cream`, and now let's specify the key, which will also delete the value with it, but let's specify the key that we want to get rid of, and let's say `weight`, and then let's `print` that again, and as you can see, the `weight` was deleted from that dictionary. So that is all we're going to cover in this data types video. Thank you guys so much for watching. I really appreciate it. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

Hello everybody. Today we're going to be taking a look at comparison, logical, and membership operators in Python. Operators are used to perform operations on variables and values. For example, you're often going to want to compare two separate values to see if they are the same or if they're different within Python, and that's where the comparison operator comes in. Right here you can see our operators; you can also see what they do. So this `==` stands for equal; we have the `!=`, the `>`, `<`, `>=`, and `<=`. And honestly, I use these almost every single time I use Python, so these are very important to know and know how to use. So let's get rid of that really quickly and actually start writing it out and see how these comparison operators work in Python. The very first one that we're going to look at is `==`. Now you can't just say `10 = 10`. Let's try running that really quickly by clicking Shift+Enter; it's going to say "cannot assign to literal". That's because this is like assigning a variable; we're trying to say `10 = 10`, and then we can call that 10 later, but that's not how this actually works. What we're trying to do is to determine whether 10 is equal to 10. So we're going to say `==`, and then if we run that by clicking Shift+Enter again, it's going to say `True`. Now if we put something else like 50 in there and we try to run this, it's going to say `False`. So really what you're going to get when you use these comparison operators is either a `True` or a `False`. If we take this right down here, we can also say `!=`, and we're going to use an exclamation point, equal sign, and that says 10 is not equal to 50, and that should be `True`. You can also compare strings and variables. So let's go right down here, and we're going to say `"vanilla" != "chocolate"`, and when we run this, it'll say `False`. Now if it was the same, just like when we did our numbers, it should say `True`. And we can also compare variables. So we'll say `x = "vanilla"` and `y = "chocolate"`, and then when we come down here, we can say `x == y`, and it'll give us a `False`, and we say `x != y`, and it'll give us a `True`.

The next one that we're going to take a look at is the `<`. So let's copy this one right up here; let's scroll down, and let let's say `10 < 50`. Now this will come out as `True`. Now let's say we put a 10 in here. Before, 10 was of course less than 50, but is 10 less than 10? No, that's `False` because they are the same. So if we want an output that is `True`, all we would have to add is an equal sign right here, and this would say 10 is less than or it is equal to 10, and now it's `True`. Of course, we can say the exact same thing by saying `>`. So `10 >= 10`; that'll be `True` because 10 is equal to 10. We can also say `50 >= 10` because 50 is obviously greater than 10.

Now let's look at logical operators that are often combined with comparison operators. So our operators are `and`, `or`, and `not`. So if you have an `and`, that returns `True` if both statements are true. If it's `or`, only one of the statements has to be true. And the `not` basically reverses the result. So if it was going to return `True`, it would return `False`. I don't use this `not` one a lot, but I will show you how it works. So let's actually test that out. So before we were saying `10 > 50`, and of course this returned `False`. So now let's add a parenthesis around this: `(10 > 50)`, and we're going to say `and`, we'll do an open parenthesis: `(50 > 10)`. Now this statement right here is `True`: `50 > 10`. So we have a `True` statement and a `False` statement, but this `and` is going to look at both of them, and it's going to say they both need to be `True` in order to return a `True`. So let's try running this, and we still have a `False`. If we want it to return `True`, we're going to have to change this to make it a `True` statement. So `70 > 50 and 50 > 10`. When we run this, it should return `True`.

Now let's look at the `or`. So let's copy this, and we'll say `10 > 50 or 50 > 10`. Now this is a `False` statement, and this is a `True` statement.

So, if even one of them is a true statement, the output should be true. And again, we can do this even with strings. So we can do vanilla and chocolate; there we go. And vanilla is actually greater than chocolate because V is a higher number in the alphabetical order. So V is like 20 something, whereas chocolate is three, right? So it actually looks at the spelling for this. So if we say "or" here, it will come out true. And if we say "and" here, it should also be true because V is greater than C and 50 is greater than 10, so this should also be true.

Now, let's copy this right here, and we're going to say "not." So what we had before is 50 is greater than 10; that returned true, but now all we're doing is putting "not" in front of it, so instead of returning true, it's going to return false.

So now let's take a look at membership operators. We use this to check if something—whether it's a value or a string or something like that—is within another value or string or sequence. Our operators are "in" and "not in." So it's pretty simple: if it's in, it's going to return true; if the sequence with a specified value is present in the object, just like we were talking about. And for "not in," it's basically the exact same thing: if it's not in that object.

So let's start out by taking a look at a string. We're going to say ice_cream is equal to "I love chocolate ice cream," and then we're going to say "love in ice_cream," and that will turn true. So all we're doing is searching if the word "love"—or that string—is in this larger string. We could also just do that by literally copying this and putting this where this is, so we can check, "Is this string part of this string?" and it'll say true. We can also make a list, so we'll say scoops is equal to, and then we'll do a bracket, and we'll say 1, 2, 3, 4, 5, and then we'll say "two in scoops." So all we're doing is searching to see if two is within this list, and that should return true. Now, if we put a six here and we said "not in," it will also return true because six is not in scoops, and that is true. And just like we did, we could also say wanted_scoops and we'll say eight, so I wanted eight scoops. So we can say "wanted_scoops in scoops," and this should return true because there's not an eight within the scoops that we wanted. And if we said "in" and we said, "We wanted eight, is that within our list that we created?" and that's going to return a false.

So that is a quick breakdown of comparison, logical, and membership operators. I hope that this was helpful. Thank you guys so much for watching. If you like this video, be sure to like and subscribe, and I will see you in the next video.

[Music]

Hello everybody. Today we're going to be taking a look at the if statement within Python. Now, it's actually the if/elif/else statement, but that's a mouthful, so I'm just going to call it the if/else statement. Now we have this flowchart, and I apologize for it being blurry, but this is the absolute best one that I could find. Right up top, we have our if condition. Now, if this if condition is true, we're going to run a body of code, but if that condition is false, we're going to go over here and go to the elif condition. The elif condition or statement is basically saying, "If the first if statement doesn't work, let's try this if statement." If this elif statement is true, it goes to this body of code; if it's false, it'll come over here to the else. And the else is basically, "If all these things don't work, then run this body of code." Now you can have as many elif statements as you want, but you can only have one if statement and one else statement.

So let's write out some code and see how this actually looks. Let's first start off by writing "if"; that is our if statement. And now we have to write our condition, which is about to be either met or not met. So we'll say, "if 25 is greater than 10," which is true, we'll say colon, and then we're going to hit enter, and it's going to automatically indent that line of code for us. And this is our body of code. So if 25 is greater than 10, our body of code will execute. So for us, we're just going to write "print," and we'll say "it worked." Now, if we run this, it's going to check, "Is 25 greater than 10?" If that is true, print this. So let's hit shift enter, and it worked.

Now let's take this—this exact code—we'll paste it right down here, and we'll say "is less than." And right now, this if statement is not true, so it's not actually going to work. As you can see, there's no output; there's nothing that happened really, but it did check to see if 25 was less than 10, but it just wasn't true. Now we can use our else statement. So we're going to come right down here, and we're going to say "else," and we'll do a colon, and we'll hit enter again, automatically indenting, and we're going to say "print," and we're going to say "it did not work..." So what it's going to do is it's going to come up here and check, "Is 25 less than 10?" No, it's not, so this body of code is not going to be executed; it's going to go right down to this else statement. Now this else statement is going to be printed; there's no condition on this. So the if statement has a condition: 25 is less than 10. This has no condition, so if this doesn't work—if this is false—it's going to come down here, and it will run this body of code. Let's run this by clicking shift enter, and as you can see, our output is "it did not work."

Now let's go back up here and put "greater than" because this is now true. It's going to say, "If 25 is greater than 10, print 'it worked'," and then it's going to stop; it's not going to go to this else statement at all. So let's run this, and our output is "it worked."

So what if we have a lot of different conditions that we want to try? Let's come right down here; this is where the elif comes in. So really quickly, let's change this to a not true, a false statement. We're going to go down and say "elif," and we're going to say, "if it is," and let's say 30. We'll say "elif worked." So now it's going to check, "Is 25 less than 10?" No, it's not. Let's look at the next condition: "Is 25 less than 30?" And if it is, we'll print "elif worked." So let's try running this, and "elif worked." Now we can do as many of these elif statements as we want; we can do—let's just try a few of them right here—so we'll say, "if 25 is less than 20," "is less than 21," and let's do 40, and let's do 50. So we'll say "elif," "elif2," "elif3," and "elif4." Now, if you look at this, the first one that is actually going to work is this 25 to 40 right here. Once this one is checked and it comes out as true, none of the other elif or else statements will work. So let's try this one; it should be elif3, and this one ran properly.

Now, within our condition so far, we've only used a comparison operator; we can also use a logical operator like "and" or "or." So we can say, "if 25 is less than 10," which it's not, let's say "or," actually, "and," we'll say "or 1 is less than three," which is true. If we run this now, it will actually work. So we can use several different types of operators within our if statement to see if a condition is true or not, or several conditions are true. There's also a way to write an if/else statement in one line if you want to do that. So we can write "print," we'll say "it worked," and then we'll come over here and say, "if 10 is greater than 30," and then we'll write "else print," and we'll say "it did not work," just like we had before, except now it's all occurring on one line. So let's just try this and see if it works. So it's saying, "print 'it worked' if 10 is greater than 30," which it wasn't, so it went to the else statement, and then it printed out our body right here. Although we didn't have any indentation or multiple lines, it was all done in one line.

Now there's one other thing that we haven't looked at yet, uh, and I'm going to show it to you really quickly, and that's a nested if statement. So when we run this, it's going to say "it worked." It works because it says, "25 is less than 10 or 1 is less than 3." Since this is true, it's going to print out "it worked," but we can also do a nested if statement. So we can do multiple if statements as well. So we're going to hit enter, and we'll say "if," and we'll do a true statement here. So we'll say, "if 10 is greater than 5," let's do a colon, hit enter, then we'll say "print," and then we'll type a string saying, "This nested if statement oops worked." Now let's try this out and see what we get. So it went through the first if statement; it said it was true, and it prints out "it worked." This is still the body of code, so it goes down to this next if statement, and it says, "If 10 is greater than 5, we're going to print this out," and you can do this on and on and on; it can basically go on forever, and you can create a really in-depth logic, and that actually happens a lot when you start writing more advanced code.

So I hope that this was helpful. I hope that you understand the if/else statement better. I hope that you understand how nested if statements work as well. Thank you guys so much for watching. If you like this video, be sure to like and subscribe below, and I'll see you in the next video.

[Music]

Hello everybody. Today we're going to be learning about for loops in Python. The for loop is used to iterate over a sequence, which could be a list, a tuple, an array, a string, or even a dictionary. Here's the list that we'll be working with throughout this video, and I have this little diagram right here which kind of explains how a for loop works. The for loop is going to start by looking at the very first item in our sequence or our list, and that's going to be our one right here. It's going to ask, "Is this the last element in our list?" and it is not, so it's going to go down to this body of the for loop. Now we can have a thousand different things that can happen in the body of the for loop, as we're about to look at in just a second. Then it's going to go up to the next element and ask, "Is this the last element reached?" So it'll be no again because we'll be going to the two, and then the three, and then the four, four, and the five. Once it reaches the five, it'll go to the body, the for loop, and then when it asks if that's the last element, the answer would be yes because it's iterated through all the items within the list, and then we would exit the loop, and the for loop would be over.

Now that may not have made perfect sense, but let's actually start writing out the syntax of a for loop so we can understand this better. To start our for loop, we're going to say "for," and then we're going to give it a temporary variable for this for loop. So it's a variable; as it iterates through these numbers, it's going to assign the variable to that number. So for this one, we're just going to say "number" because it's pretty appropriate; these are all numbers. And then we're going to say "in integers." Now, right here, you can put just about anything; this could be the list, this could be a tuple, this could be a string even, but that is what we're going to iterate through. So we're saying, "For the variable, each of these numbers within this list of integers," and then we're going to write a colon. This is the body of code that's going to actually be executed when we run through and iterate through our list. So for our first example, we're going to start off super simple, and all we're going to do is say "print" open parenthesis and say "number." As it iterates through the 1, 2, 3, 4, and 5, "number" becomes our variable that is going to be printed. So during that first loop, our one will be printed because that will be assigned right here, then through the next iteration, the two will be assigned, and it'll be put right here in each loop until the very end. So let's hit shift enter, and as you can see, it did exactly that.

Now, in this body—and I'll copy and paste this down here—in this body, we really can do just about anything we want; we don't even have to use this variable "number" right here. We can just print "yep" if we wanted to, and what it's going to do is for each iteration—all five of those—every time it loops through, it's going to print off "yep." So let's hit shift enter, and it printed off for us. So really, we weren't even using the numbers within the list; we were really just using it as almost a counter.

Now let's copy this "integers" once again. Let's go right up here, and let's go copy this for loop that we wrote. Now we do not have to call this "number"; this can be anything you want, any variable name that you'd like to name it. We could call it "jelly," and we can do "jelly + jelly." I think you're getting the picture, right? When it loops through that one, it's doing 1 + 1; when it loops through the two, it's doing 2 + 2. That is basically how a for loop works.

Now, for a dictionary, it's going to handle it a little bit differently. So let's create a dictionary really quickly. So we'll say "ice_cream_dictionary is equal to," we're going to do a squiggly bracket, so we're going to say "name," and we're going to say colon; we need to assign our value for that item, so we're going to say "Alex Freeberg." We'll do our next one separated by a comma, and we'll say "weekly_intake," and I'll say "five scoops per week." The next one we will do is "favorite_ice_creams," and for this one, we're going to do something a little bit different. For this, we're going to have a list within this dictionary, so we'll say within our list of my favorite ice creams, we'll say "mint chocolate chip," and I'll just do MCC for that, and we'll separate out by a comma, and we'll say "chocolate." So now we have this dictionary, ice_cream_dict, and within it we have my name, my weekly intake, and my favorite ice creams with a list in there as well. Let's hit shift enter, and now we're going to start writing our for loop. Now the for loop is going to look very similar, but to call a dictionary, it's just a little bit different. So we're going to say, "for ice_cream in ice_cream_dictionary.values," and then we're going to do parentheses and then a colon. Now we're going to print the ice_cream. So in order to indicate what we actually want to pull, we have to specify within the dictionary what we want; are we pulling the item? Are we pulling the value? We need to specify this; that's why we have this dot values right here. So let's run this and see what we get. So as you can see, we are pulling in the values right here; that's why we're pulling in "Alex Freeberg," five, and "mint chocolate chip," "chocolate."

Now we are able to call both of those—both the key and the value. So let's go right down here, and we can do both the key and the value, so we can pull two things at one time, and we're going to do this by saying ".items." So we could also do ".key" if we just wanted to do a key, but we want to do ".items," so we want to do both of them. So we're going to go right down here and say, "for key and value in ice_cream_dictionary.items," "print," and let's write "key," and then we'll do a comma, and then let's give it a little arrow or something like that—something like this—and then we'll do a comma, and we'll say "value," and let's print this off and see what we get. So it's looping through, and for each key and value, it's saying, "Here is the key," so that's the name; then we have "weekly_intake," then we have "favorite_ice_creams." It's giving us a little arrow, and then we're also printing off the value. So we have "name: Alex Freeberg," "weekly_intake: 5," "favorite_ice_creams: mint chocolate chip and chocolate."

So now let's talk about nested for loops. We've looked at for loops; we understand how they work and why they do what they do, but what about a nested for loop—a for loop within a for loop? For this example, let's create two separate lists. Let's create "flavors," and let's make that a list by making it a bracket, and we'll do "vanilla," the classic "chocolate," and "cookie dough"—all great flavors. So that's our first list, and then we're going to say "toppings," and we'll do a bracket for that as well, and we'll say "hot fudge," and then we'll do "Oreos," and then we'll do "marshmallows"—is how you spell marshmallows? I think it's an "e"; that looks wrong. I might be spelling it wrong, but that's okay. So let's save this by clicking shift enter, and now we have our flavors and our toppings. So now let's write our first for loop. So we're going to say "for one," as in our number one for loop, we're going to say "in flavors," and we'll do a colon; we'll click enter. Now we can write our second for loop. So we're going to say "for two in toppings," and then we'll do a colon and enter, and then we're going to say "print," and we'll do an open parenthesis, and then we're going to say "one," so printing the one in flavors, and then we're going to say "one, topped with, two." So what this is essentially going to do is we're going to say, "For one," we're going to take the very first one in flavors, and then we're going to loop through all of two as well. So we're going to loop through "hot fudge," "Oreos," and "marshmallows," and once we print that off, then we will loop all the way back to flavors and look at the next iteration or the next sequence within the first for loop. So let's run this really quickly and see what we get. So as you can see, it goes "vanilla, vanilla, vanilla," and "vanilla is topped with the hot fudge, the Oreos, and the marshmallows," and then we start iterating through our second one in our first for loop. So there's that hierarchy; so we're iterating completely through this one before we actually go to the very first for loop and start iterating through that one again. Now that is essentially how a nested for loop works. These nested for loops can get very complicated; in fact, for loops in general can get very complicated, the more you add to it and the more you're wanting to do with it, but that is basically how a for loop and a nested for loop works. Thank you guys so much for watching; be sure to like and subscribe below, and I'll see you in the next video.

[Music]

Hello everybody. It's—today we're going to be taking a look at while loops in Python. The while loop in Python is used to iterate over a block of code as long as the test condition is true. Now the difference between a for loop and a while loop is that a for loop is going to iterate over the entire sequence regardless of a condition, but the while loop is only going to iterate over that sequence as long as a specific condition is met. Once that condition is not met, the code is going to stop, and it's not going to iterate through the rest of the sequence. So if we take a look at this flowchart right here, we're going to enter this while loop, and we have a test condition right here. The first time that this test condition comes back false, it's going to exit the while loop. So let's start actually writing out the code and see how this while loop works. So let's create a variable; we're just going to say "number is equal to one," and then we'll say "while," and now we need to write our

Condition that needs to be met in order for our block of code beneath this to run, so we're going to say while number is less than five. And then we'll do colon, enter. And now this is our block of code; we're going to say print, and then we'll say number.

Now what we need to do is basically create, create a counter. We're going to say number equals number + 1. If you've never done something like this, it's kind of like a counter; most people start it at zero. In fact, let's start it at zero. And then each time it runs through this while loop, it's going to add one to this number up here, and then it's going to become a one, a two, a three each time it iterates through this while loop. Now once this number is no longer less than five, it'll break out of the while loop and it will no longer run. So let's run this really quick by hitting shift enter. So it starts at zero, and it's going to say while the number is less than five, print number. So the first time that it runs through it is zero, and so it prints zero, and then it adds one to number, and then it continues that while loop right here, and it keeps looping through this portion; it never goes back up here to this line of code. This is just our variable that we start with, and then once this condition is no longer met, once it is false, then it's going to break out of that code.

Now that we basically know how a while loop works, let's look at something called a break statement. So let's copy this right down here, and what we're going to say is if number is equal to three, we're going to break. Now with the break statement, we can basically stop the loop even if the while condition is true. So while this number is less than five, it's going to continue to loop through, but now we have this break statement, so it's going to say if the number equals three, we're going to break out of this while loop. But if this is false, we're going to continue adding to that number just like normal. So let's execute this. So as you can see, it only went to three instead of four like before, because each time it was running through this while loop, it was checking if the number was equal to three, and once it got to three, this became true, and then we broke out of this while loop.

The next thing that I want to look at, and we'll copy this right down here, is an else statement. Much like an if statement, but we can use the else statement with a while loop, which runs the block of code, and when that condition is no longer true, then it activates the else statement. So we'll go right down here, and we'll say else, and we'll do a colon and enter, and then we'll say print, and we'll say no longer less than five. Now because this if statement is still in there, it will break. So let's say six, and then we'll run this. And so it's going to iterate through this block of code, and once this statement is no longer true, once we break out of it, we're going to go to our else statement. Now as long as this statement is true, it's going to continue to iterate through, but once this condition is not met, then it will go to our else statement, and we'll run that line of code. Now the else statement is only going to trigger if the while loop no longer is true. If we have something like this if statement that causes it to break out of the while loop, the else statement will no longer work. So let's say if the number is three, and we run this, the else statement is no longer going to trigger, so this body of code will not be run.

Now the next thing that I want to look at is the continue statement. If the continue statement is triggered, it basically rejects all remaining statements in the current iteration of the loop, and then we'll go to the next iteration. Now to demonstrate this, I'm going to change this break into a continue. So before when we had the break, if the number was equal to three, it would stop all the code completely. But when we change this to continue, which we'll do right now, what it's going to do is it's no longer going to run through any of the subsequent code in this block of code; it's just going to go straight up to the beginning and restart our while loop. So what's going to happen when we run this is it's going to come to three; it's going to become three, and it's going to continue back into the while loop, but it's never going to have that number changed to be added to one to continue with the while loop. This will basically create an infinite loop. Let's try this really quickly, and as you can see, it's going to stay three forever. Eventually, this would time out, but I'm just going to stop the code really quick. So if we just change up the order of which we're doing things, we're going to say there, and we're going to put this down here. So what it's going to do now, instead of printing the number immediately and then adding the number later, we're going to add the number number right away, and then we're going to say if it is three, we're going to continue, and it's going to print the number. So let's try executing this and see what happens. So as you can see, we no longer have the three in our output. What it did was when we got to the number three, it continued and didn't execute this right here, which prints off that number. So that really is the basics of the while loop. I hope that this was helpful; I hope that you learned something in this video. If you did, be sure to like and subscribe below, and I'll see you in the next video. [Music]

Hello everybody, today we're going to be taking a look at functions in Python. A function is a block of code which is only run when you call it. So right here we're defining our function, and then this is our body of code that when we actually call it is going to be ran. So right here we have our function call, and all we're doing is putting the function with the parenthesis; that is basically us calling that function, and then we have our output. Throughout this video, I'm going to show you how to write a function as well as pass arguments to that function, and then a few other things like arbitrary arguments, keyword arguments, and arbitrary keyword arguments. All of these things are really important to know when you are using functions, so let's get started by writing our very first function together. We're going to start off by saying def; that is the keyword for defining a function. Then we can actually name our function, and for this one we're just going to do first_function, and then we do an open parenthesis, and then we'll put a colon. We'll hit enter, and it'll automatically indent for us, and this is where our body of code is going to go. Now within our body of code, we can write just about anything, and in this video I'm not going to get super advanced; we're just going to walk through the basics to make sure that you understand how to use functions. So for right now, all we're going to say is print; we'll do an open parenthesis; we'll do an apostrophe, and we'll say we did it. And now we're going to hit shift enter, and this is not going to do anything, at least you won't see any output from this. If we want to see the output, or we actually want to run that function—and some functions don't have outputs—but if we want to run that function, what we have to do is just copy this, going to put it right down here, and now we're going to actually call our function. So let's go ahead and click shift enter, and now we've successfully called our first function. This function is about as simple as it could possibly be, but now let's take it up a notch and start looking at arguments. So let's go right down here, and we're going to say Define number_squared; we'll do a parenthesis and our colon as well. Now really quickly, when you're naming your function, it's kind of like naming a variable; you can use something like X or Y, but I tend to like to be a little bit more descriptive. But now let's take a look at passing an argument into a function. The argument is going to be passed right here in the parentheses. So for us, I'm just going to call it a number, and then we're going to hit enter, and now we'll write our body of code, and all we're going to do for this is type print and open parentheses, and we'll say number and we'll do two two stars—at least that's what I call it—a star and a two. And what this is going to do is it's going to take the number that we pass into our function; it's going to put it right here in our body of code, and then for what we're doing, it's going to put it to the power of two. And so when the user, or you, run this and call this function, this number is something that you can specify; it's an argument that you can input that will then be run in this body of code. So let's copy this right here, and then we'll put it right down here into this next cell, and we'll say five. And so this five is going to be passed through into this function and be called right here for this print statement. Let's run it, and it should come out as I believe 25. That is my fault; I forgot to actually run this block of code, so I'm going to hit shift enter. So now we've def our function up here, and now we can actually call it. So now we'll hit shift enter, and we got our output of 25. Now in this function, we only called one argument, but you can basically call as many arguments as you want; you just have to separate them by commas. So let's copy this, and we'll put it right down here. Now we'll say number_squared_custom, and then we'll do number, and then we'll do power. So now we can specify our number as well as the power that we want to raise it to. So instead of having two, which is what you call hardcoded, we can now customize that, and we'll have power. And now when we call this function, we can specify the number and the power, and both of those will go into this body of code and be run, and we can customize those numbers. So let's copy this, and we'll say 5 to the power of three, and let's make sure I ran this. So let's do shift enter, and now we will call our function, and let's hit shift enter, and we got 5 to the power of three, which is 125. And just one last thing to mention is if you have two arguments within your function, and you are calling it right here, you have to pass in two arguments; you can't just have one. So if we have a five right here, it's going to error out; we have to specify both arguments for it to work.

Now let's take a look at arbitrary arguments. Now arbitrary arguments are really interesting because if you don't know how many arguments you want to pass through, if you don't know if it's a one, a two, or a three, you can specify that later when you're calling the argument, so you don't have to do it upfront and know that information ahead of time. So let's define our function, so we're going to say Define, and then we're say number_args, and we'll do an open parenthesis and a colon. Now within our argument right here, typically we would just specify here's what our argument will be; it will be number, or it will be a word, right? But what we're going to do is something called an arbitrary argument, so it's unknown, so we're going to put star, and then we'll say args. Now you will see something exactly like this typically if you're looking at tutorials that'll have star args in there, or if you're looking at just a generic piece of code, this is what will look like. But for us, we're going to actually put number. Again, we have the star, and then we have our arbitrary argument right here, and then we'll hit enter, and we're going to say print, open parentheses, and this is where it's going to get a little bit different. So we're going to say number, and then we're going to do an open bracket, and let's say zero, and then we'll do that times, and then we'll say number again with a bracket of one. So in a little bit, once we run this, and then we call this number_args function right here, we're going to need to specify the number zero and the number one that's going to be called. So let's go ahead and run this, and then we are going to call it, and let's say 5, 6, 128. So right up here, we did not know how many arguments we were going to pass through; it could be five, it could be a thousand, and we could also call in a tuple, and that's what this is right here; we're calling in a tuple. So what it's going to do now is when it calls this number, it's going to call the very first within that tuple, which will be that five, and then it'll also call in this number, which will be the first position, which is the six. So let's hit shift enter, and it's going to multiply these numbers together, so 5 * 6 is equal to 30. Now like I just said, this is a tuple, so we don't actually have to write out these numbers like we just did; we can pass through a tuple when we are actually calling this function. Let's do that right up here; let's just create—um—let's call it args_tuple, and we'll do open parentheses, and we'll do the same numbers; let's just copy it to make it easier. And now we've created this tuple right here, which we can then pass in, and this is a lot more handy, a lot more specific, and this is most likely how someone would do something like this. But let's now create this, and now we can copy args_tuple and pass it through. Now really quickly, this is going to fail, and I'm doing that on purpose, but I want to show you what you need to do in order to pass through this tuple. So right now it's going to say tuple index is out of range; all you have to do in order to use this is you have to specify a star before it, just like you did when you're creating your argument up here; you have to put a star in front of our tuple that we just passed through, and now let's try running this, and now it works properly.

Now the last two things that we're going to look at are keyword arguments and arbitrary keyword arguments. There are more things that you can learn and do within functions, but again, I'm just trying to teach you the basics to make sure that you understand how they work. So let's go right up here, and a keyword argument is kind of similar to this right here, and let's actually copy this and put it right down here. Now a keyword argument is very similar in that you're going to specify your arguments right here, but what we did up here—let me bring this down—when we actually called the function, what we did was we just put in a five and a three, and when we did that, it automatically assigned number to five and power to three, and that's totally fine, and you can do that, but if you want a little bit more control, you can use a keyword argument. So right here we could say power is equal to five, and number is equal to three. So I just switched it around, right? Number was assigned to five, and power was assigned to three, but I just switched it to show you how this might work. So let's run both of these, and now it's three to the power of five, which is 243. So that essentially is a keyword argument; again, it just gives you a little bit more control; you don't have to put them in specific positions like if you're just calling multiple arguments.

Now let's come right down here; we're going to create basically another custom function. Uh, so for this one, we're going to write Define number_or, and then we'll do an open parenthesis, a colon, and enter. And what this one is is this one is a keyword argument or an arbitrary keyword argument. Now to specify an arbitrary argument, all we did was a star, and then we input number, but if we're doing a keyword argument, we actually have to have two stars right here. So let's start taking a look, and again, if you're doing arbitrary, it means we don't really know how many keyword arguments we want to pass into our function, so we're just going to put star star number, and then later within our body of code and when we're calling it, we'll be able to specify it. And just like the arbitrary argument, for the arbitrary keyword argument means we really just don't know how many keyword arguments we're going to need to pass into our function. So to demonstrate this, let's write print, do an open parenthesis, and we'll say my—oops, need to do an apostrophe—my number is, we'll do just like that little space, and we'll say plus, and this is kind of where it gets a little interesting or a little bit more tricky. So we're going to say is number. So this is us calling our number, and then we're going to do a bracket, and then I'm actually going to go to calling the function; it's a little bit backward or a little bit different than what you might think, but when we're calling it, what I'm going to do is I'm going to say integer is equal to let's just do some random number. Now when we're calling that keyword within our body of code, what we're going to do is we're going to actually type out integer just like this, and this looks a little bit different, but what this allows us to do is we can put as many keyword arguments in here as we want later, and I'll show you in just a second, but for us, we're just creating this key and this value when we are calling it within the function. So now when we create this and we run this—oh, whoops, I forgot this has to be a string—um, so let's run this again. Now we will say my number is 2309, then we're going to add—we'll say plus—and this isn't going to look great, but we'll say my other number, because this will all be in the same, same line; that's okay. My other number, and then we'll say number, and we can specify again what we want in there. So now we can go down here to where we're calling it; we'll put a comma, and we'll say integer—oops—integer_two is equal to we'll do a random number, and then we'll put integer_two right here, and then we'll add plus right here so we don't error out. We'll create this; we'll run this, and as you can see, both numbers were passed through. Again, the syntax is terrible, but now you can see that you have this arbitrary keyword argument right here, and all we have to do is put number, number, and we can pass through as many of these arbitrary keyword arguments as we want, as long as we just specify within our function when we're calling it. So that's all we're going to look at in today's video on functions. There are, of course, other things that you can do within functions, and it can get a little bit more advanced, but I wanted to show you the basics, the meat and potatoes of—I definitely think you should know—in order to get started using functions. I hope that you were able to understand functions better because of this video; if you did, be sure to like and subscribe below, and I will see you in the next [Music] video.

Hello everybody, today we're going to be talking about converting data types in Python. In this video, I'm going to show you how to convert several different data types, including strings, numbers, sets, tuples, and even dictionaries. So let's start off by creating a variable; we'll say num_int is equal to 7, and we can check that data type by saying type and then inserting our variable num_int, and that will tell us that our data type for this variable is an integer. Let's go ahead and create another one; we're going to say num_string is equal to, and for this one we'll also do a seven, but let's check the type, and we'll do an open parenthesis, and we'll say the type of num_string, and that one is a

string. Now, let's say we wanted to add those. We'll say `num_sum`, so the sum of `num_or_int` plus `num_or_string`. Now, when we're adding these two values, it is not going to work. It's going to give us an error, and it's going to say unsupported operand for INT and string. So it cannot add both an integer and a string. What we need to do in order to add these two numbers is to convert that string into an integer. So let's go right up here. Let's add another cell, and let's say `num_or_string_converted` is equal to, and we want to convert it into an integer. So all we have to do to convert it into an integer is type `int`, and then we're going to say `num_or_string`. And that is as easy as it's going to get. All we have to do is say integer with our `num_string` inside of it, and then it's going to convert it. And we can even check it right after by saying `type(num_string_converted)`. Let's run this, and now we can see that it was converted into an integer. So now let's add that `num_string_converted` right here. Let's copy and replace that string with the string converted, and let's actually print out that `num_or_sum`, and it worked properly. Now, we did not specify what type of value this `num_sum` was going to be, but because it was two integers in here, it's going to automatically apply that data type of integer to that `num_sum`. Let's go right down here, and now let's look at how we can convert lists, sets, and tuples.

So now let's say we have a `list_or_type`, and that's equal to 1, 2, 3. And we can check it again by saying `type()`, and that is a list. Let's say we want to convert it to a tuple. It's fairly easy. All we're going to do is write `tuple()`, say `list_or_type`. That `list_or_type` is now going to be a tuple, and we can check that by saying `type()` and wrapping it around this tuple, and it shows us that it is converting that list into a tuple. Now, we can also convert a list into a set, but it may change the actual values within it. Let's check that out really quickly. So let's say we have this list, and let's add a few more values to this just like that. Now, let's say we want to convert it to a set. So we're going to run this, and we'll say `set(list_or_type)`, and let's try running this and see what the output is. So this is something that you really need to be aware of when you are converting data types, because `set()` does not act the same as a list. A set is basically going to take the unique values in the list and convert it to a set, and it fundamentally changes the data that was in that original list. And just to check the data type, we can say `type()`. I'm just doing this for all of them, and as you can see that is now a set.

Now let's go down here and take a look at dictionaries. Now let's say we have a dictionary called `dictionary_type`, and we'll do a squiggly bracket, and we'll say `name`, and we'll do a colon, and we'll say Alex. Then we'll do `age`, and a colon, and we'll say 28, and then we'll do `hair`, colon, and... So really quickly, let's take that `dictionary_type` and just confirm that it is a dictionary, and it is. And now what we're going to do is take a look at all of the items within that dictionary. So we're going to do `dictionary_type.items()`, open parenthesis, and this is going to show us all the items within it. Now we can also take this and look at something like the values, and when we run that, these are our values. So within our dictionary, we have items, and that's what this is right here. This is one item, and then within that we have our values, which are right here: Alex, 28, and Na. And then we have something called a key, and this is the key. The name, age, and hair are all keys, and we can look at that by saying `.keys()`. So let's say we want to take all of the keys and put that into a list. What we're going to do is we're going to take this right here, say `list()`, we'll do an open parenthesis, we'll type that in right there, so it says a list, and we're converting these keys into a list, and let's run that, and now this is a list, and let's just check the type as well, just to confirm, and as you can see it was converted properly into a list. And we can do the exact same thing with `values()`, and the values can also be converted into a list.

Now we can also convert longer strings that aren't just numbers like we did above in our very first example. So let's do `long_or_string`, and we'll say "I like to party". Now we're going to take this string and we're going to say `list(long_string)`. So we're going to convert this string into a list, and let's see what happens. So it took every single character in that string and put it into a list. And we could also do a set as well; that one's a lot shorter because it's only looking at unique values. So that is how you convert data types in Python. Thank you guys so much for watching. I really appreciate it. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

Hello everybody. Today we're going to be working on building a BMI calculator in Python. Now, before we get started, I want to show you this BMI calculator that I found online, and it shows you the basic calculation that they use, and that's the one we're going to use in this video. And they also have this calculator right down here and some ranges that we can use for our calculator as well. So for reference, I weigh about 170. I'm about 5'9". Let's calculate this. So I'm about a 25.1 BMI, which falls into the overweight category. That's unfortunate, but we can see exactly how this works and how ours should work when we actually build it. So we're going to kind of reference this throughout the video. So let's go right over here to our BMI calculator. We need to calculate weight and height, and then run this calculation right here. So let's go ahead and copy this, and we're going to put it right down here. And so now we have our calculation. So what we need is we need input from a user, and there is an `input()` function within Python that we're going to be using. So let's actually give me a few more cells. The first thing that we need to calculate is their weight. Let's type out `weight` right here. We'll say `weight` is equal to, and this is where we'll use our `input()` function. So we'll say `input()`, and when we actually run this, it's just going to give us this blank square, or a user can input something. We'll say Alex. So this is our output; is what the actual user input, and it does save it to this variable. So if we say `print(weight)`, it will still print out Alex. Now this is where we want the user to, just like we did before, where they'll input their weight. So we want to kind of give them a prompt for this. We'll put a string in here. So I'll do a double quote, and then I'll say "Enter your weight in", and we're using pounds, say "pounds: ". So now when we do this, it'll say "Enter your weight in pounds". I'll say 170, and then when we run this, it does store that. Now let's do `print(weight)`. I should have saved it `weight` again. Oops. Now it's only storing the value of 170; it's not actually storing this string right here. So that's really important for when we do our calculations later. Um, I'm going to... I'm going to save this right down here because I'm sure I'm going to use that later. Um, so we have that it's working. Now we need to also do our height. So let's copy this and put it right here, and we'll do `height`, and "Enter your height in inches". So now for this one, if we hit enter, it's actually running. Let's stop it really quick and interrupt it. Let's try running this. So it's going to say "Enter your weight in pounds". That's the first input. Say 170, and then when I hit enter, it's going to prompt me for that second input, and so in inches, 5'9" is 69 inches, and then I can hit enter again, and now we have both of our inputs. Now we need this calculation right down here, and just like that. So now we have `weight` in pounds * 703 divided by `height` in inches * `height` in inches. So we actually have `weight`, and it's already written in there, but I'm just going to do it like this. We'll do `weight` * 703, so that's pounds, our weight in pounds * 703, divided by, now we have our `height` in inches * the `height` in inches. So this is our calculation right here. So let's do this exact same thing. Let's run this, and this * of course is not going to work. Whoops. We need to do our * for both of these. All right, now this is our calculation. So let's run this. So we have 170, and that's pounds, and inches was 69. Hit enter, and it says "cannot multiply sequence by non-int of type 'str'". Ah, that's because these are being stored in strings. If right down here I do, and we'll do `type(height)`, and we run that, this is actually a string. So we want to change that because we don't need that anymore. That so we don't want it to be a string; we need those to be integers or floats, or really anything besides a string; it just needs to be numerical, uh, so integer, float, really. So let's do `int()`, and we'll wrap that `input()` in it, and we'll do the same thing for this one. Now we have an integer for our weight, an integer for our height. So now when we're running this calculation, it should work properly. Let's run this again. Our pounds are 170, our height is 69 in, and it's not giving us our output because we're not printing anything. Okay, so I just need to do `print(BMI)`. Let's try this again. 170, 69, and there is our BMI, 25.1. So it worked the exact same as this one. So they input... well, we input our height, we input our... or we input our weight, we input our height, and then it calculated our BMI. The next thing that we need to do is we need to kind of give the user some context. Is that good? Is their BMI within a good range, a bad range? We don't know, uh, so let's go ahead, and I'm going to see if I can copy this... know if this will work or not. Let's go ahead and copy this right down here. Perfect. So what we now need to do is we need to say, okay, if the user has given us this input, we want to give them... or tell them if they are a normal weight, overweight, obese, severely obese, anything like that, and we have these ranges, so that should help us out quite a bit. So let's just write our `if` statement, and then we'll include it up here, but let's go down here and we'll say `if`, and then we'll do `BMI`, and let's just say `BMI` is greater than zero. So if it's greater than zero, if they had any input where the `BMI` was not zero, which should be every time if they do it properly and they don't, you know, put a string in there or something, or type out 40, which maybe we should make a prompt for that. If that happens, then we can say... `if`, we'll do `BMI`, and now we need to give that first range, so this range right here. So if it's under 18.5, so we need to do a less than. So if it's less than 18.5, and it just says under, it doesn't say under or equal to, so I'll keep it at 18.5. So if it's under 18.5, then let's give kind of the output. We'll say `print()`, and the output, or the basically the prompt, is underweight. So we'll just say "You are underweight". And just like that, um, then we're going to pass several `elif` statements through here. Well, let's just say `else`. I guess this would be like if they are... if they don't input something properly, or something messes up. Maybe I... we could write something like, um, `print()` "Enter valid inputs" or something like this, or we can always change that. But let's really quickly... let's run this. Okay, so I'm not in that range, uh, let's make the next one, so then I can be within a certain range. Oops, and we need... we should need one more, a minimum. So we'll say `elif`, and `elif`. These next two are this 24.9. So it's going to check this one first. So if it's 18.5 or below 18.5, it's automatically going to print this one. So this next one, we don't have to do like a range or anything; we can just say if it's below... if it's between 18.5 and 24.9. So this one actually should be less than or equal to... um, this one is normal. Oh, whoops, 24.9. So this one is 24.9. This one is going to say "You are normal weight". So let's run this now. Let's see. `BMI` was 25.1. Oh, guys, I'm just messing up here. I apologize. All right, this is the one that I was part of. So now it's going to be part of the overweight crowd. Now let's run this, and now our prompt is "You are overweight", because remember the `BMI` was saved right here as 25.1. Down here, if we run through this, it's saying, "No, you're not in..." Oops, get rid of that. "No, you're not in under 18.5; you're not under 24.9; if you're under 29.9, you are overweight". So that did work properly. So that's really good, and I don't think I want this to be our output for the person because we're going to add this up here; it's just going to give us the `BMI`, and then the output is going to say "You are overweight," uh, let's make it a little... a little bit more customized. Um, I'm going to say `name = input()` and then we'll say "Enter your name". So it'll be "Enter your name". We'll do Alex. 170, 69. There's our `BMI`. Now it's going to run through this logic, or it will run through this logic in just a second when we actually finish this. So then we have 34.9, and let's do one more. Oops, and then this one's going to be for 39.9. So this one was overweight; this one is obese; severely obese. So we'll say "severely"... is that how you spell it? Severely obese. And then anything that's over that, 40 and over. So if it's not this one, anything else should be morbidly obese. So actually, this `else` statement right here should say, uh, "You are severely obese". This is going to say "morbidly obese". Now I added that `name` up here because I wanted to add that down below actually. So we're going to say, uh, `name +` and then we'll do like, um, uh, "You are underweight". So it'll be a little bit more personalized, uh, I think it'll... I think it'll be a nice touch. I really do. We'll do it like this, and we'll say "You", and let's go back and do that to all of them. And let me see if quickly I can do this... oh, whoops, what did I do? Got rid of that. `name +` "You" like that. Jeez, you guys are seeing me mess up a ton. `name +` "You", and then `name +` "You". So now let's run this, and now it's a little more personalized. It says "Alex, you are overweight". So this is all really good. Now this is an `if` statement, um, what we had done before, I think, is actually what we should put right down here. So we'll say `else`, and then if that doesn't work, we'll say... what do we say? "Enter valid input". We'll just put that, um, and let... let me see if I can test this out. Don't... I don't know if this will error out or if this will even work. Let me just see if I can mess with it and see if I can get it to work. Actually, let's copy this. We're going to copy this whole thing; we're going to include it right here, and now we have basically our entire calculator. So, um, let's run this. "Enter your name". We'll say Alex. "Enter your pounds". 170. "Enter your inches". 69. And then it's going to say 25.1. "Alex, you are overweight", and that's perfect. We could even go as far as adding like some feedback. We say "You are overweight", and then it would be a period, and we could say, um, "You need to exercise more. Stop sitting and writing so many Python tutorials". So now if we run this, we'll do Alex, 170, 69. It says "Alex, you are overweight. You need to exercise more and stop sitting and writing so many Python tutorials." Period. And that's it. This is the entire project. Um, you can go a ton farther; you can include much more complex logic; you could even build out a UI to create your own, you know, app just like this, where it has this input and this UI. You can build that out with in Jupyter notebooks with Python, um, but that's not really what this tutorial is for. This is just to kind of help you, um, think through some of the logic of creating something like this. So you know, I hope that this was helpful. I hope that this was fun. I like creating stuff like this. We have two other projects that we're going to do, and maybe I'll include more, but we have two right now that I have planned, um, and I hope those are helpful. This is probably our easiest one, and they'll get a little bit more difficult in the next projects. So I hope that this was... I hope that this was helpful, and that you can now kind of utilize those Python skills that you've been working on. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

Hello everybody. Today we're going to be creating an automatic file sorter for your files and file explorer. Now, out of all the projects that we've done in this series so far, I think this one might be the most difficult, but I also think this one is the most cool because it has some real-life applications. So without further ado, let's take a look at some files that we have right down here in my file explorer. So I have this beautiful picture of Rosie, uh, right here. This is a PNG file. I have a CSV file and a text file, and I want to sort all of them into their own folders depending on what kind of file it is. So if I go right in here and I click on this one, I go to properties, I can see that this is a PNG file. Um, if I go into this one, I don't need to, but if I go into this one, it's a CSV file, and of course this one is a text file. So I want three separate folders in here, and I want them to automatically go into those folders without me having to drag and drop and going and clicking. Now, we only have four files here, but imagine if we have thousands of files, how much time that could save us. So let's get out of here and let's start writing our code. So we're going to say `import os, shutil`. Now, `os` obviously stands for operating system. `shutil`, uh, I don't know what it actually supposed to stand for, but what it will allow us to do is do some high-level operations on our files in file explorer. So we're going to go ahead and import those, and now that we have those imported, uh, something that's going to be very important for us to have throughout this whole thing, and this is anytime I'm working with like directories or something like this, we want to get this path down. So I'm going to go ahead and copy this path, and we're just going to say `path =`, and we'll do this right here. So let's run this, and I need to put an `r` right here to make this a raw text. Um, so when you don't have the `r`, uh, it's going to read in these, you know, these backslashes and these colons and different stuff. If we do `r`, it's just going to read it in as the raw string, and that's what we want. So here's what we need to do. There... there's a few different things that have to happen when we are writing this out. One thing is is we need to go in here and we need to see this path, and we need to see are there...

Folders in here already. Um, if not, we need to create a folder, so that's one of the first things that we need to do. The next thing that we need is it needs to check each of these files individually, identify what kind of file it is, and then put it into the correct folder. So we have to create the folder, then check these, and then place it into the correct folder. So let's go right out of here.

So what we're going to start doing is we're going to start working with these paths and these directories, and some of these things you may never have seen before, but that's okay. I'll try to explain it as I go through. So the first thing that we're going to write is os.listdir, uh, and what this is actually going to do is show us all the files in there. We're going to say path, so it should show us all the files within path. And so here are our results. So we have the data professional results, fake text file, our image, and our other image. So this is actually showing us what files are in that path, and that's super important because we're probably going to have to loop through this in some way later. Um, I wrote this all out before, so I kind of remember, but I'm doing this all off top of my head, so I guarantee you throughout this I'll make some mistakes, but what we now need to do is we need to create folders or check if there's a folder and create it if it isn't there. That's um the next step that we need to take. So let's go right down here, and we want to check if this path exists already. So if that folder already exists, so we're going to say os.path.exists. So this is going to check, does this path, just like this path up here, does it already exist? And then we're going to do an open parenthesis; we'll say path, so that's our path. Now we need to add a folder name to this. Um, we could hardcode it, so we could do plus, we could say CSV files, and that could work. So it would say, does this path already exist? Exist? And we can try running this, and it's going to say false, so this doesn't already exist. But the thing is is we need to create three separate paths, so we could do this by just hardcoding it in by saying CSV files, image files, um, and text files, or we can just put this all on a list and loop through it. I think it's just going to be easier to do that, or I don't know, visually it's going to be easier. So we'll do uh folder_names, and we'll say is equal to, and we'll create a list. So I think I want to call it CSV files, comma, um, image files or PNG files, whatever you want to write, and then we'll do text files, do text files, and then we can go right down here.

Um, a little for loop, uh, I think what we'll do, well, actually let's write folder_names. Um, then we can to put something like, uh, let's write loop, why not? Um, so a little trick for the for loop is going to say for, and we'll say loop, and we'll just do a range because we want it to basically go through here; we don't want it to actually give us these file names; we just want it to count 0, 1, and 2. So if we do range from 0 to 2, Z 0, 1, 2, that should work. If we do um this, then when it loops through, it's going to call folder name and say zero, which would be CSV files, image files, and text files. Um, so let's, uh, yeah, I need a colon. Let's run through this really quickly; shouldn't do anything, but what we can do now is we can say, okay, if this does not exist, what we can do is actually create it. So we'll say if not, so if this does not exist, then what we're going to do is take this, and we'll say os.makedir, and then we'll do just like that. Um, I think it's makedirs; I can't, I think that's correct. Um, so let's test this out really quickly. Let's see if this works. Invalid syntax; I need a colon. Okay, so I just ran this; let's see if it did actually make those folders. Let's refresh it, and it didn't. So let's just print this off. Um, so if not, let's just print, let's see, does this actually work? Let's do if. Ah, okay, so I think I know what might be happening. I think it's giving us, it may actually be, let, let's check this really quick. Go to python tutorials. Oh, no, I think it's creating, yeah, it's creating these Python tutorial images right here. Whoops. Okay, so I just figured it out. Um, let's go back into python tutorials; don't take a look at any of those notebooks; those are secret. Um, we were creating them in the wrong place, um, um, and that's because of this right here; we need a backslash. So we need to actually include a backslash right here in this path; we didn't have that. Ey scanning string literal. Okay, so this backslash could cause an issue. Let's see if I can do forward slashes on all these. Just stick with me, guys; I might cut this out; I might not; we'll see if this is important. Just going to keep talking while we're doing it. Um, let's run this. Okay, so now that we're doing these forward slashes, we're still checking; let's make sure we can still check those files. Good. Now when we loop through this, I'm not going to, well, yeah, I can print it off; doesn't matter; I'm going to print it, and we'll see if that name works. And then we're also going to, uh, I said if, so if it exists, then make it, no, no, no. So if not, I think the not did make sense; we just weren't sure; we had to do some um checking. So if it exists, then we're going to create it, and we'll keep the print in there because it doesn't really matter. So it's going to create the CSV and image but didn't create the text. Let's see. Okay, let's, I don't know why this would work, but let's run it. Okay, so I think I just had the wrong range. So now we have our images all through, or we have our folders, all three folders. Now we need to write a script that will read in these and check and see what kind of file it is and place it into the correct folder. So let's come right down here, and let's see what we need to do. So now I think we need to use this right here. Um, I think we need to loop through this to be able to check each one. So we need to name this, so we'll just do um file_name is equal to, run that. So now we have this file_name, um, and what we can do is loop through this. So let's say, let's say for file in file_name, so we're going to loop through this. Now when it goes through, it needs to check the, it's going to check the file path, and in the file path it'll say .txt, CSV. So let's say, um, if I think it should be CSV. Let's test it on this one, but if CSV is in file_name, or actually it's file, so if if it's in file and not in, and oh, not not in, but if it's also not in this, I believe, because we're going to check, we're going to check each of those folders. So to loop through and it's going to check and see if the CSV, so if that string is in the file, then what we want to do is check that it's also not in here. That's actually just the folder; we also need, also we're not doing that for loop anymore. Um, okay, I'm sorry; I'm talking this through; I'm figuring it out as I go because I may have forgotten some of this. So we're going to say this, that's the CSV files, so we need to check this one. Um, let's do it like this. Oops. Okay, so it's going to check to see if CSV files, and I think it needs that in between it, so it's going to say the path, so there's our path plus slash CSV files. Um, actually, no, it needs to be like this because we're going to check that. Then I got it all right; I figured it out now. Then we're going to check if this file is in there. Yeah, so that's right. So it says if the CSV is in the file, um, which is right, where am I looking? Oh, file_name. So if it's in that list of the actual files, which is all of these, if we find CSV in any of these files and it's not already in here, so it's going to say path plus CSV files. Did I say files? Yeah, CSV files plus file. Okay, that all looks correct. So if it's not in there, we're going to use shutil.move. Now this is how we actually move the file; it gives us the ability to move what we want. Then we'll say move; we need to take it from our initial path to our new path. So we're going to specify; we'll separate by comma; we need to specify its original path, which it should just be this without this. I think it should be file, path because this is where it is now; it's in the FI, this path with that file name. Then we need to say we want to move it to here; that is what we want to do. Um, yeah, so let's check it with just this one and see if it works. Okay, it ran through it; let's go check. Aha! Now that CSV file is gone; perfect; that is exactly what we want it to happen. Now we can just recreate this for um for both our PNG files, our image files, and our text files. So we'll say elif and elif, and let's do PNG, then we'll do image files and image files because again we're just doing the exact same thing. I can do text files; the next one's going to be text files, text files. So this one's going to check for .txt. Now do we need anything else? Um, we'll just say else, and we'll print off, print this file type is not included, or or if there's multiple files, we'll say there are files in this path that were not moved. Okay, so if we run through this, it's going to catch our our CSV, catch our PNG, catch our text, and if not, it'll say there are files in this path that weren't moved, exclamation point. All right, now let's run through this. Uh, uh, that's because if, elif, elif, and then it's going to this else statement. Uh, I don't know; let's let's circle back around to that in a second. All of them were moved properly; that's really good. Really quickly, I I'll I'll check and see; I just don't, I'm going to take that out for now, so I'm just going to run it. Um, I we may or may not go back to that, but let's check and see if everything worked properly. So let's go into the CSV file, and we have our CSV file. Let's go into our image files, and we have our images. And let's go into our text file, and there are our text files. Now is there anything else that we need to do? I just don't believe so, but what I can do is I can take all this; I can include it in here, and I'm going to basically restart it just to see if it works properly from scratch, right? I just want to make sure that I didn't miss anything, and we'll delete these. So we have our, I'm just going to rerun everything. We, we imported; we created our path; these are our file names; and then when we run this, it should take our folder names, check through them; if they aren't already created, it's going to create it. Don't need it to print, so let's get rid of that. Then for the file within our file names, and it check it, it checks each one; we check if there's a CSV, and if it's already in that file, if it's already in that folder, I mean, if it's in that folder, then it doesn't do anything, but if it isn't, so and not it's not in there, it is going to move it to that location. So it's going to check CSV, PNG, and text. I think everything should work properly. Let's run this, and it looks like it's working. Good, good, good, and perfect; it worked exactly how I had hoped. Um, that's great. So this is the automatic file sorter in file explorer project. Uh, you can go even a step further. So I had to come in here and manually run this; you can go a step further and put a timer on this where it automatically does this, maybe every hour, every day, every 30 minutes. You can run this in your background, especially if you create um like a an execution for this; you can run this in your background. Um, if you are curious on how to do that, I think I did something similar to that in my web scraping project, um, my Amazon web scraping project. If you want to go check that one out, we're not going to do it in this project; this is all I wanted to show you how to do. So I hope that this was helpful; I hope that this project was, you know, interesting and that you liked it, and I hope that you learned something. And so if you did, be sure to like and subscribe below, and I will see you in the next video.

What's going on, everybody? Welcome back to another video. Today we're going to be starting our Python web scraping tutorial series. Now this is more of a continuation of the Python tutorial series, but because we're going to be focusing on web scraping for three or four videos, I wanted to just make it its own little miniseries. In this series, I'm going to show you the basics of web scraping, how to actually look at HTML, how to inspect a web page, how to pull that data in, and then even put it into a CSV file so you can save it and use it. Now in this series, we're just covering the basics, which is a fantastic place to start, but in future series I'll be going into some of the more advanced web scraping topics as well. So without further ado, let up on my screen and get started with web scraping.

Now the first thing that we need to learn is HTML. HTML stands for hypertext markup language, and it's used to describe all of the elements on a web page. Now when we actually go to a website and start pulling data and information, we need to know HTML so we can specify exactly what we want to take off of that website. So that's where HTML comes in, and we're going to look at the basics, understanding just the basic structure of HTML. Then we'll go look at a real website, and you'll kind of see that's a little bit more difficult than what we just have right here, but this is the basic building blocks to get to what the HTML actually looks like on a website. Now this is basically what HTML looks like. We have these angled brackets with things like HTML, head, title, body, and then you'll notice that at the end we'll have a body, and then we'll have a body at the bottom. This forward slash body denotes that this is the end of the body section in HTML. So everything inside of this is within this body. So there is this hierarchy within HTML. We have HTML and HTML at the bottom, which encapsulates all the HTML on the website. Then we have things like head and head, body and body. Now within these sections, we usually have things like classes, tags, attributes, text, and all these other things, things that we'll get to in different lessons, but one of the easiest ones to notice and look at are tags, things like a P tag or a title tag. Now within these tags, because this is a super simple example, we have these strings here, my first web page, and this is what's called a variable string, and this is actual text that we could take out of this web page.

Now that you understand these super basics of HTML, let's actually go to our website, and I'm going to have a link down below, but it's going to be this one right here. This is basically just a website that you can, you know, practice web scraping on; it's called scrapethissite.com. And what we're going to do is look at the HTML behind this web page, and you can do this on any website that you go on. So we're going to right-click; we're going to go down to inspect. Now right off the bat, this looks a lot more complicated and a lot more complex than the very simple illustration that we were looking at, but let's kind of roll this up just a little bit. You'll notice we have HTML and HTML at the bottom; we have a head, and there is the end of the head, and then a body and the end of the body. So in a super simple sense, it is similar, but just the information that's within it is a lot more difficult. Now if we look at this title right here, this is our title tag. If we click this little arrow, this is our drop-down; you'll notice that here we have the string hockey teams, forms, searching, pagination. Now let's say we didn't know; we didn't want to click on that and go find it. There's something that's super helpful within this inspection page that you can click on right here; it says select an element in the page to inspect it. So we're going to click on that, and as we go through our page, and let's click on this title, it's going to take us to exactly where this is in our HTML. This is extremely helpful, extremely useful. For example, let's say the data I want is down here; I want to take in the Boston Bruins; I can click on it, and it's going to take me to where that is exactly in the HTML. This is where we can start writing our web scraping script to specify, okay, I'm looking for a TR tag; I'm looking for a TD tag; I'm looking for the class called team. This is all information and things that we can use to specify exactly what we want to pull out of our web page. Now there are other things that we didn't really look at as well in just our simple illustration. Let's come right over here; there's things like HREFs. Now these, these are hyperlinks. So if we went and then clicked on this, this is just regular text, but inside of it is this hyperlink where if we click on it, it would take us to another website, and typically that's denoted by this HREF right here. Then you'll typically see things like a P tag, which usually stands for a paragraph. Now the last thing that I want to show you while we're here, and we're going to learn a lot more in the next several lessons, but if we come right down here, there is this actual entire table here, and let's try to find this table, and I'm having trouble selecting the entire thing, but let's select this team name, and if we look at this team name, you can see that this is encapsulating the table, this table tag. Now these are super helpful because it takes in the entire table. Now if we wrap this up and we look just at this, it says class table, and then we have the end of this table tag. Now when we open it, it's going to have all of this information. So as you can see, as I'm highlighting over it, we have these TH tags, and we have these TD tags, and even these TR tags, which is the individual data, and this is something that we'll look at when we're actually scraping all of the data from this table in a future lesson. So this is how we can use HTML, how we can inspect the web page and see exactly what's going on kind of under the hood, and then in future lessons we'll see how we can use this HTML to specify exactly what data we want to pull out. Thank you guys so much for watching. If you like this video, be sure to like and subscribe below. I will see you in the next lesson. [Music]

Hello everybody. In this lesson, we're going to be taking a look at Beautiful Soup and requests. Now these packages in Python are really useful; these are the two main ones that I use when I was first starting out with web scraping; it can get a lot of what you want done in order to get that information out. Now, of course, there are other packages that you can use that may be a little bit more advanced, but again, this is just the beginner series. In a future series, we'll look at other packages as well that have some more advanced functionality.

So what we're going to be doing is we're going to import these packages, and then we're going to get all of the HTML from our website and make sure that it's in a usable state. And then, in the next lesson, we're going to kind of query around in the HTML, kind of pick and choose exactly what we want. We'll look at things like tags, variable strings, classes, attributes, and more. So let's get started by importing our packages. What we're going to say is: from bs4—this is the module that we're taking from—we're going to say import, then we'll do beautiful soup. Then we're going to come down and we're going to say import requests. Now let's go ahead and run this. I'm going to hit shift enter, and it works well for me. Now, if this does not work for you, you may potentially need to actually install bs4. So you may have to go to your terminal window and say pip install bs4. I'll just let you Google how to do that if you need to do that, cuz it's pretty easy. But if you're using Jupiter notebooks through Anaconda, like how we set it up at the beginning, beginning of this Python series, then you should be totally fine; it should be there for you.

The next thing that we need to do is specify where we're taking this HTML from. So what we need to actually do is come right over here to our web page and we need to get the URL. So we're going to go here; we're going to copy this URL, and I'm just going to put it right here for a second. And what we're going to do is we're going to be using this URL quite a bit, so we just want to assign it to a variable. So just say URL is equal to, and then we'll put it right in here. Now we can get rid of that. So now this is our URL going forward; this is where we're going to be pulling data from. Let's go ahead and run this. Now we're going to use requests, and what we're going to do is we're going to say requests.get, and then we're going to put in URL. Now this get function is going to use the request Library; it's going to send a get request to that URL, and it's going to return a response object. Let's go ahead and run this. As you can see here, I got a response of 200. If you got something like a 204 or 400 or 401 or 404, all these things are potentially bad. Something like a 204 would mean there was no content in the actual web page. 400 means a bad request, so it was invalid; the server couldn't process it, and you don't get any response. If you got a 404, that might be one that you're familiar with; that's an error that means the server cannot be found.

The next thing that we're going to do is take the HTML. Now, if you remember, we come right back here, and we inspect this; we have all of this HTML right here. Now, on this web page specifically right now, it's completely static; it's not a bunch of moving stuff or anything like that. Usually, when you're looking at HTML, if you're looking at something like Amazon, and those web pages can update, but when you actually pull that into Python, you're basically getting a snapshot of the HTML at that time. So what we're going to do is bring in all of this HTML, which is our snapshot of our website, and then we can take a look at it. So we're going to come right down here, and now we're going to say beautiful soup. So now we'll use the beautiful soup package Library. So we need to say beautiful soup, and we're going to do an open parenthesis; we're going to do two things; there's two parameters that we need to put in here. First, we need to put in this get request; we actually need to name this, and we'll call this page; we'll say page is equal to, and let's run this. And now we're going to put that page in here, and what we're going to say is text. So the page is what's sending that request, and then the text is what's retrieving the actual raw HTML that we're going to be using. Then we're going to put a comma here, and what we need to specify is how we're going to parse this information. Now this is an HTML, so what we're going to do is HTML just like this. This is a standard; this is already built into this Library, so we don't need to go any further, but it's basically going to parse the information in an HTML format. Now let's go ahead and run this. Let's see what we get. And as you can see, we have a lot of information, and as we scroll down, I'll try to point out some things that we've already looked at in previous lessons—um, something like this th tag that should be very similar; that's the title. Then we have these TD tags, and then, of course, if we scroll down even further, we'll have things like ATR tag. So these are all things that we looked at in that first lesson when learning about HTML.

Now, again, we want to assign this to a variable, so we're going to say soup; that's going to say equal to this information right here. Now I'm not going to go into all the history behind beautiful soup. What I will say is the guy who created this beautiful soup Library—uh, what he said was is that it takes this really messy HTML or XML, which you can also use it for, and makes it into this kind of beautiful soup. So I just thought that was kind of funny, uh, but that's why we're calling it soup right here. And we're going to go ahead and run this, and we'll come right down here and we'll say print soup, and let's run it. And now we have everything in here. So we have our HTML, our head, we have some HR and some links in here. Let's scroll down a little bit more, and then we have our body right there, and of course, we have a bunch of information in here. Now, in the next lesson, what we're going to be doing is learning how to kind of query all of this to take specific information out and basically understand a lot of what's going on in this HTML to make sure we can actually get what we need.

Now, if this looks really kind of messy to you, and it just doesn't make a lot of sense, there is one more thing that I'm going to show you, and we'll come right down here. So we'll say soup.prettify. And if you've ever used a different type of programming languages, uh, pretty is very common in a lot of them, where it'll just make it a little bit more easy to visualize and see. Uh, you'll notice that it kind of has this hierarchy built in, whereas if we scroll up, there's no hierarchy built in; it's all just down this left-hand side. So if you kind of want to view it and just kind of visually see the differences, this does help a lot, but it doesn't actually help a lot when you're, you know, querying it or using, you know, find and find all, which is what we're going to look at in the next lesson. So that is our lesson on beautiful soup and request. In the next two lessons, we're going to be looking at find and find all, as well as really diving into things like variable strings and tags and classes and all those things. And then, in the last lesson, we're going to do kind of this mini project where we try to get all the data from this web page that we've been using from that table and put it into a panda data frame. So thank you guys so much for watching; I really appreciate it. If you like this video, be sure to like and subscribe below, and I will see you in the next lesson. [Music]

Hello everybody. In this lesson, we're going to be taking a look at find and find all. Really, we're going to be looking at a ton of different things in this lesson. This is where we really start digging in, seeing how we can extract specific information from our web page. But in order to do that, let's set everything up where we actually bring in the HTML like we did in the last lesson, and we're just going to write all this out one more time, just for practice if nothing else, and then we'll get into actually getting that information from the HTML. So we're going to start by saying from bs4 import beautiful soup. There we go. And import requests. We'll go ahead and run this. Then we're going to come up here, grab our HTML, or sorry, our URL, so we'll say URL is equal to, and we'll have that right here. Now we need to say page is equal to, and then we'll do requests.get, and then we'll put in our URL right here. And we're going to come over here and run this. And lastly, we need to say soup; we'll say soup is equal to beautiful soup. There we go. And then, within our parentheses, we need to specify the page.text because we need that and our parser, which is HTML. And there we go. And let's go ahead and run this. Let's print it out; make sure it's working. And there we go. So we have our soup right here. All this should look really similar to our last lesson. And so now we've brought in our HTML from our page; we have a lot, a lot, a lot of information in here.

Now, really quickly, let's come over and let's inspect our web page. Now, in here, we have a ton of information; right, we have a bunch of different tags and classes and all these other things, but how do we actually use these? Well, that's where the find and find all is going to come into play, and they're pretty similar, and you'll see that in just a little bit. But let's say we want to take uh one of these tags, and let's come down; let's say we just want to take this div tag. Now there's going to be a lot of different div tags in our HTML, but let's just come right here; let's go down, and let's say we're going to call Soup; we're going to say soup—that's all of our information—we're going to say dot find. Now, within our parentheses, we can specify a lot of different things, but we're going to keep it really simple right now; we're just going to say div. Let's go ahead and run this. What this is going to bring up is the very first div tag in our HTML, and that's going to be this information right here. Now let's copy this, and we're going to do the exact same thing, except we're going to say find_all. Now let's run this. Now we're going to have a ton more information. Really, all find and find all do is that they find the information. Now find is only going to find the first response in our HTML—lead; that's the div class container. Let's go back up to the top; that's our div class container. But find all is going to find all of them, so it'll put it in this list for you. So it's going to have this first one, and it goes down to uh this div, which should be right here, and then we have a comma, which separates our next div tag. So that is how we can use it.

Now, what if we want to specify one of these div tags? We pulled in a ton of them, but we want to just look for one of them. Well, this is something where the class comes in handy because right now we have classes equal to container, classes equal to co md-12. I don't know what these are at the off the top of my head, but um, usually they'll be somewhat unique, and we can use these to help us specify what we're looking for. For example, just kind of glancing at this, we could also use this a tag if we wanted to look at this. So we could say, oh, we're looking for uh these hrefs. So we have an href here, and this right down here, we have this href as well, which again, uh, if you remember from the previous lesson, that stands for a hyperlink. Now, something like the class or the href, um, or these IDs, these are all attributes, so we can specify or kind of filter down based off of these. Now let's try it. So what we can do is we can do class first, and this is kind of the default uh within something like find all; you can even do class_—come right back up—we have this div, and then here's our class. So again, we have to have the div and the class. If we took this a tag, this is an a tag, which would go right here with the class of something like navlink or something like navlink. Again, down here, we need to specify that more, but we have our div, so we'll say cl Cole md12 right here, and let's go ahead and run this. And now it's going to pull in just that information. Now we're still getting a list because we have multiple of these. So this div class uh Cole md-12 doesn't just happen once; if we scroll down, we'll see it multiple times, something like right here, uh, or actually let me see, right here. So here's this comma, then here's our next one. So we have two of these uh div tags with a class of coal-md-12, and in each of these, we have different information. This looks like a paragraph with this P tag right here. And let's scroll back up. Uh, so I also think we should try out doing something like this P tag. Typically, these P tags stand for paragraphs, or they have text information in them. Let's try to P tag really quickly and let's just see what we get. And let's run this, and it looks like we get multiple P tags. Now, if we come back here, you can see that there's this information, and it's this information that we're pulling in, and I'm just, you know, noticing that from right here, and then we have this information right here. It looks like there's one more, which is this href, which looks like this open source, so data via, and then that uh hyperlink or that link right there. So we have three different P tags. Now, just to verify and make sure that that's correct, what we could do is come over here; we're going to click on this paragraph; it's going to take us to that P tag where the class is equal to lead. Let's come over here and look at this paragraph. Now we have another P tag right over here with the class is equal to glyphicon glyphicon SL education. I have no idea what that means, um, and then we'll go to our last one, which is right here, where the P tag is equal to—we have a tag H class—and a bunch of other information. So let's say we just wanted to pull in this paragraph right here. Let's go here and see how we can specify this information. So it looks like P, or the class is equal to lead; that looks like it's going to be unique to just that one. So if we come down here, we're going to say comma, and it was class, so you can do uh class underscore is equal to, and then we're going to say lead. Let's try running this, and we're just pulling in that information.

Now let's say we actually want to pull in this paragraph; we actually want this text right here, and this is a very real use case. You know, let's say I'm trying to pull in some information or or a paragraph of text. Well, let's copy this, and what we're going to then do is say .text, and let's run this. Now we're going to get an error right here, and this is a very common error because we're trying to use find all. Unfortunately, find all does not have a text attribute; we actually need to change this to find. Typically, when I'm working with these find and find alls, I'm using find all most of the time until I want to start extracting text; then, when I specify it, I'll change this back to find just like this. Now let's try this, and now we're getting in parentheses this information. Now this is all wonky; it needs to definitely be cleaned up a little bit, but if we code back up, it's no longer in a list, and we no longer have things like these P tags in here or this class attribute, so we're really just trying to pull out this information. Now, again, this does not look perfect; we could even try to do something like strip; look like there's some white space—uh, that cleans it up a little bit. This definitely looks a little better, um, and we could definitely go in here and clean this up more, but just for, you know, an example, this is how we can then extract that information.

Now let's look at one more example. This is some information, and this is what we're going to do, kind of our little mini project in the next lesson on—let's say we wanted to take all this information. Well, what if we wanted to pull in something like the team name? That's going to be in right here in this TR tag, and each of these TR tags have th tags underneath them. So if we scroll down, you'll notice that each row is this TR tag. So let's go ahead and search for—let's do th; let's just search for that first. So let's come right back up here; let's use this find all, and we'll get rid of this text for right now, and let's just say we want to look for the TR; is that what we said we were looking for? No, th. So let's say we're looking for th. Let's go ahead and run this. So we're going to have underneath this th; we have team name, year, wins, losses, and notice these are all the titles. So these titles are the only ones with these th tags. If we go down, you'll notice that the data is actually TD tags. So now let's go back and look for TD; we'll say td, and this is going to be a lot longer; we have a lot of information, but these are all the rows of data. Let's see if we can just get one piece of this data; we're going to get back; we want just this team name; that's all we're trying to pull in for now, um, and then we'll try to get this row, and then in the next lesson, we're going to try to get all of this information, make it look really nice, and then we'll put it into a panda's data frame. So let's just get this team name right now. Let's go ahead; we're going to say th; let's run this, and we have this th, and now that we know we're getting this information in, we can do find. Let's run this; there's our team name. I'm just going to say .text, and again, we can do .strip just like that, and bam, we have our team name. So you can kind of start getting the idea of how we're pulling this information out; we're really just specifying exactly what we're seeing in this HTML, and what's really, really helpful, and you know, something that I do all the time is I'm inspecting it; I'm just kind of searching, like, like how, what do I want? What piece of information do I want? Then I go ahead and click on it, and then I'm looking, you know, where is this sitting in the hierarchy? It's within the body; it's within this table with the class of table, then it's down here where this TR tag and then this TD tag. So I'm looking kind of at the hierarchy, and I'm specifying exactly what I'm looking for. So that is what we're going to look at in today's lesson; that's how we can use find and find all. We were able to look at classes and tags and attributes and variable strings, which—which is this right here—getting that text uh and variable strings, and we will look at find and find all and how it's pulling that information in and how we can specify exactly what we're looking for. Now, in the next lesson, which is definitely going to be the most exciting one, we're going to try to pull in all of this information, so every single thing, because we'll be able to put all this information into a data frame, which then we can use pandas to really search and manipulate that data within that data frame. So, with that being said, that is the end of this lesson. If you like this video, be sure to like and subscribe; I will see you in the next [Music] lesson.

Hello everybody. In this lesson, we are going to be scraping data from a real website and putting it into a panda's data frame and maybe even exporting it to CSV if we're feeling a bit spicy. Now, in the last several lessons, we've been looking at this page right here, and I even promised that we were going to be

Pulling this data; but as I was building out the project, I just—I honestly thought it was a little bit too easy. Since in the last lesson we kind of already pulled out some information from this table, and I want to kind of throw you guys off, so we're going to be pulling from a different table. We're going to be going onto Wikipedia and looking at the list of the largest companies in the United States by revenue, and we're going to be pulling all of this information. So if you thought this was going to be easy in a little mini project, uh, it's now a full project, because uh, why not? So let's get started.

Uh, what we're going to do is we're going to import beautiful soup and requests. We're going to get this information, and we're going to see how we can do this, and it's going to get a little bit more complicated and a little bit more tricky. We're going to have to, you know, format things properly to get it into our Panda data frame to make it looking good and making it more usable. So let's go ahead and get rid of this easy table; we don't want that one. Uh, and we're going to come in here, and we're just going to start off—this should look uh really familiar by now. We're going to say from bs4 import BeautifulSoup. I don't know if you've noticed, but I've messed up spelling BeautifulSoup in every single uh video I've noticed. Uh, let's run this, and now we need to go ahead and get our URL. So let's come up here; let's get our URL. Say URL is equal to, and we'll just keep it all in the same thing really quickly because we know this by heart by now, right? Uh, we'll say requests.get and then URL to make sure that we're getting that information; it'll give us a response object. Um, hopefully it'll be 200; that'll mean a good response. And then we'll say soup is equal to, and then we'll say BeautifulSoup, and we'll do our page.text. Now we're pulling in the information from this URL, and then we use our parser, which will be oops, HTML. And let's go ahead and run this. Looks like everything went well. Let's print our soup. Now this is completely new to you; it's completely new to me; I don't know what I'm doing, uh, but it looks like we're pulling in the information, am I right? So we got a lot of things going for us. Uh, the uh stuff was imported properly; we got our URL; we got our soup, which is uh not beautiful in my opinion, but let's keep on rolling. Let's come right down here. Now what we need to do is we need to specify what data we're looking for. So let's come and let's inspect this web page. Now the only information that we're going to want is right in here; we're going to want these uh titles or these headers. Whoops. So we're going to want Rank, Name, Industry, etc., and then we are for sure going to want all of this information. Let's just scroll down; see if there's anything tricky in here. All right, that looks pretty good. Uh, and there is another table; so there's not just one table in here; there are two tables in this page, so that might change things for us, but let's come right back and let's let's inspect our page by using this little button right here, and let's specify in—let's see if I can highlight just this page—oh, it's not—oh, let's do that right there. So now we have this uh Wiki table sorter. Now I'm going to actually come right here; I'm going to copy and I'm just going to say copy the outer HTML; I'm just going to paste it in here real quick, and that's a ton of information; I didn't think it was going to copy all of it, and we're just going to delete that. I just wanted to keep that class uh because I wanted to then come right down here at the bottom and just see what this table uh looks like. I don't know if it's part of it or if it's a if it's its own table. Um, I can't tell. Let's look at this Rank and let's come up; so it says uh it's under this table, and it looks like it's its own table, but it says Wiki table sort sortable jQuery table sorter. What could be a sortable jQuery table sorter? So it looks like there are two tables with the same class, which shouldn't be a problem if we're using find to get our text because we should be taking the first one, which will be this table, and this is the table we want. Um, and if we wanted this one, we could just use find all, and since it's a list, we could use indexing to pull this table right. Um, but I think we're going to be okay with just pulling in this one. So let's go ahead and let's do our find. So we'll do soup.find, and we could find all or we could just do find uh table. Let's just try this and see what we get, and if it pulls in the right one that we're looking for, that'd be great. Now this does not look correct at all. Um, I don't know what table it's pulling in. Oh, maybe it's this right here; this might be a table. Yeah, it is. So we have this uh box more citations. So actually we are going to have to do exactly like what I was talking about. Uh, let's pull this, and we—well, we could do comma class uh right here, and let's do both. You know what, this is a learning opportunity; let's do both. So let me go back up to the top because I need these, um, and what we're going to do—let come right down here—I want to add in uh another thing; actually, I'll just push this one up there. We go. So we're going to say find all. Let's run this. So now we have multiple, and again we got that weird one first, but if we scroll down, here's our comma, and then here's our Wiki table sortable, and then we have Rank, Name, Industry—all the ones that we were hoping to see—and I guarantee you if we scroll all the way to the bottom, um, we're going to see potentially Wells Fargo, Goldman Sachs. I'm pretty sure those are—let's see—yeah, here we go: like Ford Motor, Wells Fargo, Goldman Sachs. That's this table right here. So now we're looking at the third table, but again this is a list, so we can use indexing on this, and we'll just choose not position zero because that's this one right here, which we did not like. Well, now we'll take position one. Let's run this. Let's go back up to the top, and this is our table right here: Rank, Name, Industry. This is the information that we were actually wanting. Just to confirm: Rank, Name, Industry, etc. So this is the information we're wanting, and we're able to specify that with our find all, and this is the information we want. So we now want to make this the only information that we're looking at. So I'm just going to copy this. We didn't need to use our class for this one; you could probably could have, um, but we could. So let's actually um put this right down here. This will be our table; we'll say equal to, but then I'll come right here, and I'm going to say soup.find. This is just for demonstration purposes. We do table, comma, class is equal to, and then we'll look at this right here. Whoops, do this, and let's see if we get the correct output, and let's run this, and looks like we're getting a NoneType object. Uh, if I remember, looks like the actual class is this right here. So let's run this instead, and I got to get rid of the index. There we go. Okay, so we were able to pull it in just using the find, so the find table class and it says Wiki table sortable; at least that's the HTML that we're pulling in right here. Let me go back because I don't I don't know if that's what I was seeing earlier. Let's just get this Rank; let's go back up. Oh, where's the Rank go? Rank, there we go. So here's our Rank, and let's go up to the table, and there's our class. Yeah, and and that's just—to me that's a little bit odd. So it says Wiki table sortable jQuery-table-sorter right here, but in our actual—in our actual Python script that we're running, it was only pulling in the Wiki table sortable; so it wasn't pulling in the jQuery-table-sorter. Why? Uh, I'm not 100% sure, but all things that we're working through, and we were able to—we were able to figure out. So we're going to make this our table; we're going to say tables equal to uh soup.findall, and let's run this, and if we print out our table, we have this table. Now this is our only data that we are looking at. Now the first thing that I want to get is I want to get these titles or these headers right here; that's what we're going to get first. So let's go in here; we can just look in this information; you can see that these are with these th tags, and we can pull out those th tags really easily. Let's come right down here; we're just going to say th, and we can get rid of this. Let's run this. Now these are only th tags because everything else is a tr tag for these rows of data. So these th tags are pretty unique, which makes it really easy, which is really great because then we can just do world_titles is equal to. So now we have these titles, but uh they're not perfect, but what we're going to do is we're going to loop through it. So I'm going to say world_titles, and I'll kind of walk through what I'm talking about. Is in a list, and each one is within these th tags, so th and then there's our um string that we're trying to get. So we can easily take this list and use list comprehension, and we can do that right down here. So I'm going to keep this where we can see it. Um, we'll do world_titles that's equal to. Now we'll do our list comprehension; should be super easy. Uh, we'll just say for title in world_titles, and then what do we want? We want title.text; that's it. Um, because we're just taking the text from each of these; we're just looping through, and we're getting Rank, then we're looping through getting Name, looping through getting Industry; that's it. So let's go and print our world_table_titles and see if it worked, and it did. Uh, this looks like it needs to be cleaned up just a little bit. So let's go ahead and do that while we're here before we actually put it into the uh Panda's data frame. Oops, I just wanted—I just wanted this actually. So what we're going to do is try to get rid of those backslash ends. If we do do strip, that may actually not work. Yeah, uh, because this is a list; what we need to do is we can actually do it dot text.strip right here. Let's try to do it in there. There we go. So now we have uh this, and now this world_tables is good to go. Now I'm actually noticing one thing that may be odd. Yeah, so we have Rank, Name, Industry, goes to Headquarters, but then in here we're getting Rank, Name, Industry, and then the Profits, which is from this table right here, which we don't want. Uh, let's scroll back up; let's kind of backtrack this and see where this happened. We did find all table; we're looking at the first one right, and then we're doing [Music] Headquarters. Uh, so we're doing print table. Ah, okay, I think I found the issue here, and let's backtrack again. This is—we're working through this together; we're going to make mistakes. Uh, the table is what we actually wanted to do; we just did soup.findall th, which is going to pull in that secondary table. Um, jeez, we were not thinking here. Um, so now we need to do find all on the table, not the soup, cuz now we were looking at all of them. Oh, what a rookie mistake. Okay. Uh, let's go back. Now let's look at this. Now it's just down to Headquarters. Okay. Okay, let's go ahead and run this. Let's run this. Now we just have Headquarters. Now let's run this. Now we are sitting pretty. Okay, excuse my mistakes. Hey, listen, you know, if it happens to me, it happens to you; I promise you this is—you know, this is a project, this a little—a little project we're creating here, so we're going to run into issues, and that's okay; we're figuring out as we go. Now what I want to do before we start pulling in all the data is I want to put this into our Panda data frame; we'll have the uh, you know, headers there for us to go, so we won't have to get that later, and it just makes it easier uh in general; trust me. So we're going to import pandas as pd. Let's go ahead and run this, and now we're going to create our data frame. So we'll say pd. Now we have these world uh table titles, so what we're going to do is pd.DataFrame, and then in here for our columns, we'll say that's equal to the world_table_titles, and let's just go ahead and say that's our data frame and call our data frame right here. Let's run it. There we go. So we were able to pull out and extract those headers and those titles of these columns; we're able to put it into our data frame, so we're set up and we're ready to go; we're rocking and rolling. The next thing we need—let's go back up—next thing we need is is to start pulling in this data right here. So we have to see how we can pull this data in. Now if you remember that we have those th tags, those were our titles, as you can see I'm highlighting over it, but down here now we have these td tags, and those are all encapsulated within a tr tag. So these tr represent the rows, right? Then the td represents the data within those rows. So r for rows, td for data. So let's see how we can use that in order to get the information that we want. So let's go back up here; just going to take this because again we're only pulling from table, not soup, not soup; what were we thinking? Um, and let's go ahead and let's look at tr. Let's run this. Now when we're doing this tr, these do come in with the headers, so we're going to have to—later on we're going to have to get rid of these; we don't want to pull those in um and have that as part of our data, but if we scroll down, there's our Walmart. Um, we have the location; these are all with these td tags, and then of course it's separated by a comma, then we have our td2. So above we had our td1, so Row one, Row two, Row three all the way down. Now we will easily be able to use this right because this is our column data, and we can even call it that. column_data is equal to. We'll run that, um, and what we're going to do is we're going to loop through that. CU it was all in a list, so we're going to loop through that information, but instead of looking at the tr tag, we're going to look at the td tag. So let's come right down here; we'll say for the row in column_row, and we'll do a column. Now we need to loop through this; we'll do something like row.findall, and then what are we looking for? We're not looking for the tr; looking for the td, and just for now let's print this off—off—see what this looks like. Apparently I didn't run this uh column_data; that's why. And let's run this, and what we actually need to do is something almost exactly like this, and I'm going to put it right below it. Um, instead of printing this off because again this is all in a list; we're using find all, so we're—we're printing off another list, which isn't actually super helpful. Um, for each of—or all these data that we're pulling in, what we can do is we can call this uh the row_data, and then we'll put the row data in here. So we'll say for and we'll say in row_data, so we'll just say for the data in row_data, and we'll take the data; we'll exchange that, and now instead of uh world_table_titles, we can change this into uh individual_row_data right, and now let's print off the individual_row_data. So it's the exact same process that we were doing up here, and that's how we cleaned it up and got this, and we may not need to strip, but let's just run this and see what we get. There we go. Um, and strip I'm sure was helpful. Let's actually get rid of this. Yeah, strip was helpful; it's the exact same thing that happened on the last one, so let's keep that. Actually, let's run this, and now let's just kind of glance at this; let's look through it. This looks exactly like the information that's in the table. Let's just confirm with this first one: uh, 25 uh two—what am I saying?—572,754, 2.4, 2300, 57275, 2.4, 2200. So this looks exactly correct. Now we have to figure out a way to get this into our table because again these are all individual lists; it's not like we're just, you know, putting all this in it one time; we can't just take the entire table and plop it into—into the data frame; we need a way to kind of put this in one at a time. Now if you're just here for web scraping and you haven't taken like my Panda series, that's totally fine; that's not what we're here for anyways. Um, but what we can do—we'll have our individual_row_data, and we're going to put it in kind of one at a time. Now the reason we have to do that is because when we had it like this, and let's go back, when we had it like this, it's printing out all of it, but what it's really, really doing, and let's get rid of it, um, what it's really doing is it's kind of doing it like this; it's printing it off one at a time, and it's only going to save that current row of data; this last one, it's only going to save that as it's looping through. So what we actually want to do is every time it loops through, we append this information onto the data frame. So as it goes through, and eventually it's going to end up with this one, but as it goes through—let's run this—as it goes through, it puts this one in, and then the next time it loops through, it puts this one in, and the next time it loops through, etc., all the way down. Um, so let's see how we can do this. So we have our data frame right here. Let's get rid of this; let's bring our data frame in. Now again, like I just mentioned, if you don't know pandas and you haven't learned that, uh, you know, go take my uh series on that; it's really good. And we do something very similar to this in that Series, so I'm not going to kind of walk through the entire logic, um, but there is something called loc, which name for location when you're looking at the index on a data frame, and we're going to use that to our advantage. So we're going to say the length of the data frame, so we're looking at how many rows are in this data frame, and then we're going to say that's our length; then we're going to take that length and use it when we're actually putting in this new information; pretty—pretty cool. So we're going to say df.loc, then a bracket, and we're putting in that length, so we're checking the length of our data frame each—each time it's looping through, and then we're going to put the information in the next position; that's exactly what we're doing. Let's go ahead and put in the individual_row_data. Um, so let's just recap: We're looping through this tr; this is our column_data, so these tr, that's our row of data; then we're—as—as we're looping through it, we're doing find all and looking for td tags; that's our individual data, so that's our row_data; then we're taking that data, each piece of data, and we're getting out the text, and we're stripping it to kind of clean it, and now it's in a list for each individual row; then we're looking at our current data frame, which has nothing in it right now; we're looking at the length of it, and we're appending each row of this information into the next position. So let's go ahead and run this; it's working; it's thinking.

And it looks like we got an issue: andot set a row with mismatched columns. Now we're encountering an issue, not one that I got earlier, but we're going to cancel this out. We're going to figure this out together. So let's print off our individual row data. Let's look at this; this one is empty. Uh, this is—I'm almost certain—is probably the issue. Um, I didn't encounter this issue when I wrote these, uh, when I wrote this lesson, um, but I'm almost certain that this is the issue right here. So let's do the column data, but let's start at position, um, let's try one, and, and not parentheses; I need brackets because this is a list, right? So it should work, and there we go. So now that first one's gone, so now we just have the information. I didn't even think about that, um, just a second ago, but I'm glad we're running into it, in case you ran into that uh issue. Let's go ahead and try this again, and it looked like it worked. So let's pull our data frame down. I could have just wrote DF. Let's pull our data frame down, and now this is looking fantastic now. Um, these three dots just mean there's information in there; it just doesn't want to display it, but it looks like we have our rank, we have our name, have the industry, revenue, revenue growth, employees, and headquarters for every single one. So this is perfect. Now this is exactly what I was hoping to get. Now you can go in and use pandas and manipulate this and change it and, you know, dive into all the information in there, but we can also export this into a CSV if that's what you're wanting. So we could easily do that by saying we'll do DF.to_csv, and then within here we're just going to do R and specify our file path. So let's come down here to our file path, then we'll go to our folder for our output. So we're just going to take this path, and let me do it like that, so I have this path in my OneDrive Documents Python Web Scraping folder for output. So, you know, I already made this, um, and I'm just going to put this right down here. Now I do have to specify what we're going to call this; we'll just call this companies, and then we have to say CSV. That is very important. Now if we run this, I already know, just because uh we have this Rank and this index here, we're going to keep this index in the output; not great, uh, but let's run it. Let's look at our output; there's our companies, and when we pull this up, as you can see, this is not what we want because we have this extra thing right here. Now if we're automating this, this would get super annoying. So what we're going to do is go back and just say index_col=False. Let's go out of here, and now we're just going to come right down here; we're going to say comma index=False, and so it's going to take this index and it's not going to import or actually export it into the CSV. Now let's go ahead and run this. Let's pull up our folder one more time, and let's refresh just to make sure; should be good. And now this looks a lot better. So we're able to take all of that information and put it into a CSV, and it's all there. So this is the whole project. So if we scroll all the way back up, let's just kind of glance at what we did here. Scroll down; we brought in our libraries and packages; we specified our URL; we brought in our soup, um, and then we tried to find our table. Now that took a little bit of uh testing out, but we knew that the table was the second one, so in position one. So we took that table; we were also able to specify it using find, but then we use the class, and of course we just wanted to work with that table; that's all the data we wanted. So we specified this is our table, and we worked with just our table going forward. Of course, uh, we encountered some small issues—user errors on my end—but we were able to get our world titles, and we put those into our data frame right here using pandas. Then next we went back, and we got all the row data and the individual data from those rows, and we put it into our panel data frame. Then we came below, and we exported this into an actual CSV file. So that is how we can use web scraping to get data from something like a table and put it into a pandas data frame. I hope that this lesson was helpful. I know we encountered some issues; that's on my end, and I apologize, but if you run into those same issues, hopefully that helped, uh, but I hope this was helpful, and if you like this, be sure to like and subscribe below. I appreciate you; I love you, and I will see you in the next lesson [Music].

So the first thing that we need to do is import our pandas Library. So we're going to say import, and we're going to say pandas. Now this will import the pandas library, but it's pretty common place to give it an alias, and as a standard when using pandas, people will say as pd. So this is just a quick alias that you can use, uh, that's what I always use, and I've always used it because that's how I learned it, and I want to teach it to you the right way, so that's how we're going to do it in this video. So let's hit Shift+Enter. Now that that is imported, we can start reading in our files. Now right down here, I'm going to open up my file explorer, and we have several different types of files in here: we have CSV files, text files, JSON files, and an Excel worksheet, which is a little bit different than a CSV. So we're going to import all of those; I'm going to show you how to import it, as well as some of the different things that you need to be aware of when you're importing. So we're going to import some of those different file types, and I'll show you how to do that within pandas. So the first thing that we need to say is pd. and let's read it a CSV because that's a pretty common one; we'll say read_csv, and this is literally all you have to write in order to call that in. Now it's not going to call it in as a string like it would in one of our previous videos if you're just using the regular operating system of Python. When you're using pandas, it calls it in as a data frame, and I'll talk about some of the nuances of that. So let's go down to our file explorer; we have this countries of the world.csv. You just need to click on it and right-click and copy as path, and that's literally going to copy that file path for us, so you don't have to type it out manually; you can if you'd like, and we're just going to paste it in between these parentheses. Now if we run it right now, it will not work. I'll do that for you; it's saying we have this Unicode error. Uh, basically what's happening is is it's reading in these backslashes and this colon and all those backslashes in there and this period at the end. What we need to do is read this in as a raw text, so we're just going to say r, and now it's going to read this as a literal string or a literal value and not as, you know, with all these backslashes, which does make a big difference. When we run this, it's going to populate our very first data frame. So let's go ahead and run it, and now we have this CSV in here with our country and our region. Now if we go and pull up this file, and let's do that really quickly, let's bring up this countries of the world; it automatically populated those headers for us in the data frame, but we don't have any column for those 0, 1, 2, 3. So if we go back, as you can see right here, there's this index, and that's really important in a data frame; it's really what makes a data frame a data frame, and we use index a lot in pandas. We're able to filter on the index, search on the index, and a lot of other things which I'll show you in future videos, but this is basically how you read in a file. Now if we go right up here in between these parentheses and we hit Shift+Tab, this is going to come up for us. Let's hit this plus button, and what this is is these are all the arguments or all the things that we can specify when we're reading in a file, and there are a lot of different options. So let's go ahead and take a look really quickly. Really quickly, I wanted to give a huge shout out to the sponsor of this entire Panda series, and that is Udemy. Udemy has some of the best courses at the best prices, and it is no exception when it comes to pandas courses. If you want to master pandas, this is the course that I would recommend; it's going to teach you just about everything you need to know about pandas. So huge shout out to Udemy for sponsoring this Panda series, and let's get back to the video. The first thing is obviously the file path; we can specify a separator, which there is no default. So when we're pulling in the CSV, when we're reading in the CSV, it's automatically going to assume it's a comma—CU—it's a comma-separated uh file. You can choose delimiters, header names, index columns, and a lot of other things, as you can see right here. Now I will say that I don't use almost any of these; the few that I'm going to show you really quickly in just a second are up the very top, but you can do a ton of different things, and I'm just going to slowly go through them. So that's what those are. You can also go down here; this is our docstring, and you can see exactly how these parameters work; it'll show you and give you a text and walk you through how to do this. Again, most of these you'll probably never use, but things like a separator could actually be useful, and things like a header could be useful because it is possible that you want to either rename your headers or you don't have a header in your CSV, and you don't want it to autop populate that header. So that is something that you can specify. So, for example, this header one, and I'll show you how to do this; the default behavior is to infer that there are column names if no names are passed. This behavior is identical to header=0, so it's saying that first row or that first index, which it's like right here, that zero is going to be read in as a header, but we can come right over here, and we'll do comma header=, and we can say None, and as you can see, there are no headers. Now instead it's another index, so we have indexes on both the x-axis and the y-axis, and so right now we have this zero and one index indicating the first column and the second column. If we want to specify those names, we can say the header=None, then we can say names=, and we'll give it a list, and so the first one was country, and what's that second one? Oh, region. So right here, that's the first, um, the first row, but we'll rename it, and we'll just say country and region, and when we run that, we've now populated the country in the region; we're just pretending that our CSV does not have these values in it, and we have to name it ourselves. That's how you do it, but let's get rid of all that because we actually do want those in there, so we're just going to get rid of those and read it in as normal, and there we go. Now typically when you're reading in a file, what you need to do is you want to assign that to a variable; almost always when you see any tutorial or anybody online or even when you're actually working, people will say df=, df stands for data frame. Again, this is a data frame. In the next video in this series, I'm going to walk through what a series is as well as what a data frame is because that's pretty important to know when you're working with these data frames, but we'll assign it to this value, and then we'll say we'll call it by saying df, and we'll run it, and that's typically how you'll do things because you want to say this data frame, so later on you can do things like data frame., and you can, uh, you know, pass in different modules, but you can't really do it; it's not as easy to do it if you're calling this entire CSV and importing it every time. So let's copy this, this because now we're going to import a different type of file.

So now we've been doing read_csv, but we can also import text files. Now you can do that with the read_csv; we can import text files. Let's look at this one; we have the same one; it's countries of the world, except now it's a text file because I just converted it for this video. I'll copy that as a path, and so now when we do this, oops, let me get those quotes in there, it'll say world.txt; it will still work. Work, as you can see, this did not import properly; um, we have this country\SLT region, and then all of our values are the exact same with this \SLT. That's because we need to use a separator, and I'll show you in just a little bit how we can do this in a different way, but with that read_csv, this is how we can do it; we'll just say sep=, we need to do \SLT. Now let's try running this, and as you can see, it now has it broken out into country and region. We could also do it the more proper way, and this is the way you should do it, and I'll get rid of these really quickly, but just want to keep them there in case you want to see that, but you can also do read_table, and let's get rid of this separator, and now we have no separators; it's just reading it in as a table. Let's run this, and it reads it in properly the first time. This read_table can be used for tons of different data types, but typically I've been using it for like text files; um, we can also read in that CSV, so let's change this right here to CSV; we can read it in as a CSV, but just like we did in the last one when we read in the text file using read_csv, this read_table, you're going to need to specify the separator. So just copy this, and we'll say comma, and now it reads it in properly. Again, you can use that for a ton of different file types, but you just need to specify a few more things if you don't want to use the more specific read_uncore function when you're using pandas. Now let's copy this again; we're going to go right down here, and now let's do JSON files. JSON files usually hold semi-structured data, um, which is definitely different than very structured data like a CSV where has columns and rows. So let's go to our file explorer; we have this JSON sample; we will copy this in as path; let's paste it right here, and we'll do read_json. Again, these different functions were built out specifically for these file types; that's why, you know, each one has a different name. So now we're reading this in as the JSON; let's read it in, and it read it in properly. Now let's go ahead and copy this and take a look at Excel files because Excel files are a little bit different than other ones that we've looked at, um, so let's just do read_excel, and let's go down to our file explorer, and let's actually open up this this workbook. As you can see, we have Sheet1 right here, but we also have this world population which has a lot more data. Let's say we just wanted to read in Sheet1; we can do that, or by default it's going to read in this world population because it's the first sheet in the Excel file. Well, let's go ahead and take a look at that; let's get out of here, and let's say oops, I forgot to copy the file path; let's go ahead and copy as path, and we'll put it right here, here, and let's just read it in with no arguments or anything in there or no parameters. When we read it in, it's reading in that very first sheet, so this is the one that has all of the data. Now let's say we wanted to read in that extra sheet name or the second sheet name; we'll just go comma sheet_name=, and then we can specify sheet—was it Sheet1? Like this? Yes, it was. So we just had to specify the sheet name right here, and then it brought in that sheet instead of the default, which is the very first sheet in that Excel. Now that definitely covers a lot of how you read in those files. Again, you can come in here and hit Shift+Tab and this plus sign and take a look at all the documentation, and you can specify a lot of different things—things that I didn't think were very important for you guys to know, especially if you're just starting out. The ones that we looked at today are what I would say are like the ones that I use almost all the time, so I wanted to show you those, but if you're interested in any of these other ones or you have very unique data and you need to do that, um, you know, it's worth really getting in here and figuring things out. A few other things that I wanted to show you just in this kind of first video or this intro video on how to read in files, um, one thing that you may have noticed, especially in this file right here, is we're only looking at the first five and then the last five. So if we wanted to see all the data, all the data is in these like little three dots right here, right? We want to be able to see that data, but right now we can't, and that's because of some settings that are already within pandas, and all we need to do is change that. So this one has 234 rows and four columns, so obviously we can see all the columns. Well, let's just change the rows; all we'll say is pd.set_option. Now what we need to do is we're going to change the rows; we're not going to change the columns, at least not on this one, so we'll say 'display.max_rows'. Now if we just run this for whatever data we bring in, it's going to be able to show the max rows, and then we'll say 2035, although there's 234 rows; I'm just going to be safe. Let's run this, and now it has changed it. So let's read in this file again, and you'll see how it's changed. Now we have all the numbers, and we have this little bar on the right that allows us to go down all the way to the bottom and all the way to the top. So now we can actually look and kind of skim and see our values. I like that better than just having that, you know, shorter version; um, we can do the exact same thing on columns as well. So if we look at this one, this is our JSON file; has the same thing right here; we have what was it? 38 columns, but we can only see—I think it's maybe—it's 20 or something like that; I can't remember, um, but we have 38; we can only see like, let's say 15 of them or 20 of them. We'll do the exact same thing, and we'll just say pd.set_options.max_columns, and we'll set that to 40 for that one. When we run this, oops, let's get over here; when we run this one again, we can now scroll over and see every single one of our columns. Now that one is, in my opinion, a lot more useful; I like being able to see every single column, so definitely something that you should be using, especially when you have these really large files; you want to be able to see a lot of the data, a lot of the columns, so when you're slicing and dicing and doing all the things that are about to learn in this Panda series, you know, you know what you're looking at. I also want to show you just how to kind of look at your data in

These data frames as well; that's also pretty important. So let's go right down here, and the very last one that we imported was this one right here: this read Excel. So this data frame is the only one that's going to read in. Let's run it. Um, this is the last one to be run, so this variable right here, DF, it won't be applied to all these other ones. Um, which we can always go back and change those. Typically, you'll do something like data Frame 2; you want to do something like that. Um, so let's keep data Frame 2. Oops. So what we're going to do is we're going to bring data Frame 2 right down here, and we want to take a look at some of this data. We want to know a little bit more about it. Something that you can do is data frame 2.info, and we'll do an open parenthesis. And when we run this, it's going to give us a really quick breakdown of a little bit of our data. So we have our columns right here: rank, CCA 3, country, and capital. It's saying we have 234 values in those columns because there's 234. Scroll up here because there's 234 rows. That tells me that there's no missing data in here, at least not, you know, completely missing like null values. There is something in each of those rows. The count tells me it's non-null, so there's no null values, and it tells me the data type. So it's ringing in as an integer, an object, an object, and an object. Object and it also tells us how much memory it's using, which is also pretty neat. CU. When you get really, really large data types, memory usage and and knowing how to work around that stuff does become more important than when you're working at these really small, you know, sample sizes that we're looking at. We can also do—oops, let me get rid of that—can also do data frame 2, and we'll do shape, and for this one we do not need the parentheses. And all this is going to tell us is we have 234 rows and four col columns. We're also able to look at the first few values or rows in each of these data frames, so we can just say dataframe 2.head. And if we do that, it's going to give us the first five values, but we can specify how many we want. You can say head 10; it'll give us the first 10 rows right here. We can do the exact same thing, and let's go right down here, and we'll say tail, so they'll give us the last 10 rows within our data frame.

Now let's copy this, and let's say we don't want to actually look at all of these values or all these columns. We can specify that by saying df2 and—oops, let's get rid of all of this—and we'll say, with a quote, we'll say Rank, and now we can take just a look at the rank data. Now we can't do that by doing the index, or at least not like this. If we want to use this index that is right here, we can, but there's a very special function called loc, and I look for that, and I'm going to have an entire video on this because it does get a little bit more complex, but there's df2, and there's loc, and loc stands for location. Location; that's only for the indexes, whether it's the x axis or the y axis; those are the indexes. And for location, it's looking for the actual text, the actual string of the index. So if we come up here, that data Frame 2, we can specify 224, and it'll give us this information right here: little, little different format. So let's go bracket, and we'll say 224, and when we run this, it gives us our rank, CCA, country, capital with our values over here, kind of like a dictionary almost. Now let's copy this, and we'll say df2.iloc. And right now these look the exact same, but we haven't really talked a lot about changing the index, and you can change the index to a string or a different column or something like that, and we'll look at that in future videos. The iloc looks at the integer location, so even if these—let's go right up here—even if this index had changed to, let's say, this rank or this CCA 3 or country or whatever you make this index, the iloc will still look at the integer location, so that 224 would still be 224 even if it was Uzbekistan. So then when we look at this, it's going to be the exact same, but if we had changed that index, this loc is the one that we could search on, and we could search Uzbekistan—is that how you spell Uzbekistan? Hey, I nailed it! So that is how you use loc and iloc. Again, I just wanted to show you a little bit about how you can look at your data frame or search within your data frame. Now, in future videos, I'm going to dive a lot deeper into a lot of the concepts that we just looked at because I just kind of touched on them. I wanted you to have a brief introduction to them so that in future videos I'm not just dropping everything on you all at once. So hopefully this was a good quick introduction to those top topics. Uh, you should be able to read in a file now, see your data frame, and kind of look at it in a few different ways that we just looked at, and I hope that that was helpful. And if it was, be sure to check out all my other videos on Python and pandas, and if you like this video, be sure to like and subscribe below, and I will see you in the next [Music] video.

Hello everybody. Today we're going to be looking at filtering and ordering data frames in pandas. There are a lot of different ways you can filter and order your data in pandas, and I'm going to try to show you all of the main ways that you can do that. So let's kick it off by importing our data set. So we're going to say data frame is equal to, and we'll say pandas, and I need to import my pandas, so we'll say import pandas as pd. That's pretty important, I think. Um, so pd.read_csv, and we'll do r, and then we'll say a world pop population.csv. So let's run this. All our data frame right here, and this is the data frame that we're going to be filtering through and ordering in pandas. So let's kick it off. The first thing that we can do is filter based off of the columns, so the data within our columns: Asia, Europe, Africa, or whatever data we may have in that column. Let's go right down here. We're going to say DF, and then within it we're going to specify what column we're going to be filtering on. Say DF with another bracket, and we'll say rank. So we're going to be looking at this rank column right here, and then we'll say in that rank column we want to do greater than 10, and that's actually going to be a lot of them. Let's do less than. So when we run this, it's only going to return these values that are less than 10. We can also do less than or equal to, you know, all of these comparison operators, so less than or equal to. So now we have all of the ranks 1 through 10. Now if we look at these countries, we can specify by specific values almost exactly like we did here, but instead of doing a comparison operator like we did right here and including those names, let's say Bangladesh and Brazil. We can use the is in function, almost like an in function in SQL, if you know SQL. So let's go right down here, and we're going to say specific_countries. So right now we're just going to make a list of the countries that we want, and then we'll say Bangladesh and Brazil. So let's go right down here, and we'll say, okay, for these specific countries from the data frame, let's do our bracket. We'll say in this country column, so we'll do data frame and then another bracket for Country. So in this country column, we can do .isin and then an open parenthesis and then look for our specific countries. So looking at just this column, and we're saying is in, so we're looking at are these values within this column, and we're getting this error, and this looks very, very odd. Let me—this doesn't look right. There we go. I just had some syntax errors. I apologize. Made it way more complicated than it needs to be, but here's how you use this is in function. So we're looking at Bangladesh and Brazil, and we return those rows with Bangladesh and Brazil. Really quickly, I wanted to give a huge shout out to the sponsor of this entire Panda series, and that is Udemy. Udemy has some of the best courses at the best prices, and it is no exception when it comes to pandas courses. If you want to master pandas, this is the course that I would recommend. It's going to teach you just about everything you need to know about pandas. So huge shout out to Udemy for sponsoring this Panda series, and let's get back to the video. We can also do a contains function, kind of similar to is in, except it's more like the like in SQL as well. I'm comparing a lot of this to SQL because when you're filtering things, I always—my brain always goes to SQL—but in pandas it's called the contains. So let's do—let's actually copy this because I don't want to make the same mistake again. Let's do that, and we'll do the bracket, but instead of .isin, we're going to do .str.contains and then an open parenthesis. So we're going to be looking for a string if it contain—if it contains—let's do United, almost like United States or or any other United. So let's run this, and as you can see, we have United Arab Emirates, United Kingdom, United States, United States Virgin Islands. So we can kind of search for a specific string or a number or a value within our data or within that column of country now. So far we've only been looking at how you can filter on these columns. We can also filter based off of the index as well, and there's two different ways you can do it, or two the main ways: there's filter, and then there's loc and iloc. Loc stands for location, and iloc stands for integer location. And if you've seen other previous videos, I've kind of mentioned those. So we can take a quick look at all of those. So really quickly, we need to set an index because the index right now is uh not the best. We'll set our index to Country. So let's say df2 is equal to DF.set_index, and we'll say country. I'm just doing df2 because later on I want to use that data frame again, so I'm just going to assign it to another data frame so that we can just easily switch back and forth. So now we have this index as the country, and what we can do is use the filter function. So let's go down here. We'll say df2.filter, and we'll do an open parenthesis, and now we can specify our items. So these are actually going to be specifying which columns we want to keep. So we're going to say items is equal to, then we'll make a list. We'll say continent—hope that's how we spell continent; I'm always messing up with my uh my stuff here, my spelling—then we'll do CCA 3 because why not? You can specify whichever ones you want. When we run this, it's going to only bring in those two columns. Now, by default it's choosing the axis for us, but we can also specify which axis we want to search on. So if we say axis is equal to 0, it's actually going to search this axis; this is the 0 axis, this is the 1 axis. So where our columns are is 1. So if we go back and do 1, we're searching on that 1 axis, or those header axes again, and this is the default, but you can specify that. So if you just want to search on uh, you know, filtering right here, you can do that. And let's actually copy this and do that right down here just so you can see what it looks like, but let's search for Zimbabwe, and we'll do Zimbabwe, and we'll be looking at the 0 axis, which—which is the up and down on the left-hand side. And when we filter on that, we can filter by Zimbabwe by looking just at the country index. We can also use the like just like we did before, and I'll show you the exact same demonstration that we did, which you can say like is equal to, and instead of having to put in a concrete um text, you can just say United just like we did before, and we're searching where the axis is equal to 0, which again is this left-handed axis. So now we're looking for United, and it's going to give us all of the countries or all the indexed values that have United in it. Like we were talking about before, we also have loc and iloc. So we can say data frame 2.loc. Now this is a specific value, so we'll do United States. So location is just looking at the actual name or the value of it, not its position. So if we search for United States, it's going to give us this right here where it gives us all of the column for United States and then all of the values for United States. Or we can do the iloc, which is the integer location, which is not the exact same because we're looking at the string for the loc; we're looking at this string, but underneath it there still is a position; that's that integer location. Let's do a completely random one; let's just say 3. If we look at the third position, it's going to give us ASM, which—which I'm not exactly sure what it is, but it still gives us basically the same kind of output, which is the columns and the values. So that's another way that you can search within your index when you're actually trying to filter down that data.

Now let's go look at the order by, and let's start with the very first one that we looked at. Let's do data frame; that's why I kept it because I wanted to use it later. Now we can sort and order these values instead of it just being kind of a jumbled mess in here. We can sort these columns however we would like: ascending, descending, ing multiple columns, single columns, and let's look at how to do that. So we'll say data frame, and then we'll do data frame. Look at rank again, just like we were doing above, and let's do data frame where it's less than 10. I should have just gone and copied this; I apologize. So now we have this data frame that is greater than 10. Now we can do .sort_values, and this is the function that's going to allow us to sort everything that we want to sort. So we can do by is equal to, and we'll just order it by the exact same thing that we were doing or calling it on; we'll do rank. So now what this going to do, it's going to order our rank column, and as you can see, it did that: 1, 2, 3, 4, 5. We can also do it with ascending or descending. So if you want to, you can look in here and see what you can do. So we'll do ascending; we'll say that's equal to true, and so that's the automatic default, so that didn't change anything, but if we say false, it's going to be descending from highest to lowest. So now we have it in the opposite direction. Now we don't have to just order or sort this on one single column; we can do multiple columns, and we can do that by making a list right here—whoops—make a list just like that, and we'll input different ones as well. So now let's input our country, and when we run this, it will give us rank of 9, 8, 7, 6, as well as the country of Russia, Bangladesh, Brazil. Now if you noticed, the country really didn't change because the rank stayed the exact same; that's because there's an order of importance here, and it starts with the very first one. If we change this around and we look at this one and put a comma right here, now the country is going to be descended, and the rank would come second. So it's not—the rank isn't going to really have any effect here. So now we have the country: United States, Russia, Pakistan, and the rank really didn't get ordered at all. Now if we want to see how that can actually work, let's do continent right here and actually put it right here and do country here. So if we run this, it's first going to come, and it's going to organize or sort the continent, then it's going to come back and go to the country, and then it's going to sort the country. So keep—so keep your eye right here in this Asia area because we're going to sort this differently than ascending. So we have ascending false, and that applies to both of these; it's false and false, but we can specify which one we want to do. We can do a false here and a true here. So we'll do false, comma true, and what this is going to do is it's going to say false for the continent, so the continent right here is going to stay the exact same. And so that is a lot of how you can filter and order your data within pandas. I hope that this was helpful. I hope that you enjoyed this video. If you liked it, be sure to like and subscribe below, check out all my other videos on Python and pandas, and I will see you in the next [Music] video.

Hello everybody. Today we're going to be looking at indexing in pandas. If you remember from previous videos, the index is an object that stores the access labels for all Panda objects. The index in a data frame is extremely useful because it's customizable, and you can also search and filter based off of that index. In this video, we're going to talk all about indexing, how you can change the index and customize that as well as how you can search and filter on that index, and then we're also going to be looking at something a little bit more advanced called multi-indexing, and you won't always use it, but it's really good to know in case you come across a data frame that has that in it. So let's get started by importing pandas: import pandas as pd. Now we'll get our first data frame: we'll say DF is equal to pd.read_csv, and I've already copied this, but we're going to do r, and we're going to put this file path. So I have this world population.csv; I will have that in the description just like I do in all of my other videos. Let's run DF, and let's take a look at this data frame. So we have a lot of information here: we have rank, country, continent, population, as well as the default index from zero all the way up to 233. Now, if you haven't watched any of my previous videos on pandas, the index is pretty important, and it's basically just a number or a label for each row. It doesn't even necessarily have to be a unique number. Um, you can create or add an index yourself if you want to, and it doesn't have to be unique, but it—it really should be unique, uh, especially if you want to use it appropriately for what we're doing. The country is actually going to be a pretty great index because the country, you know, is going to be all unique because we're looking at every single row as a different um country, country as well as the population. So let's go ahead and create this country or add this country as our index. Now we can do this in a lot of different ways, but the first way that you can do this if you already know what you are going to create that index on is we can just go right in here when we're reading in this file, and we'll say comma index_—oops, I spelled that completely wrong—index_col, and we'll say that is equal to, and then we're going to say quote country. So we're taking this country and we're going to assign it as the index. Now let's read this in, and as you can see, this is our index now. It looks a little bit different; we didn't have this country header right here, which is specifying that this is still the country, but you can tell that this is the index based off the um bold letters as well as it being on the far left, and all the regular columns for the data is over here while the country header is right here, and it's lower than all the others; just a quick way that you can see that.

That is the index. Now, before we move on, I want to show you some other ways that you can do this as well. But I'm going to show you how to reverse this index before we move on. And we'll say data frame. So we had our data frame right here, so we have data frame dot. We'll say reset index and then we'll say in place is equal to True, which means we don't have to assign this to another variable and all that stuff; it'll just be true. So now, when we run that data frame again, the index was reset to the default numbers.

So now let's go down here, and I'll show you how to do this in a different way. You can do DF dot. We'll say set index, and then we'll just say country. So very similar to when we were reading in that file and we said set the index or that index column, we said index column equals country. If we do this and we run it, it works. But if we say data frame right down here, it's not going to save that. If we want to save it just like we did above, we're going to say in place is equal to true. True. That is going to save it to where we don't have to assign it another variable. So now, when we run this, the data frame right here, which is going to populate this, the data frame is going to say in place is equal to true, so that country will now be our index again. Let's run this, and there we go.

Now, what's really great about this index is we're able to search based off just this index, and so we can filter on it and basically look through our data with it. And there are two different ways that you can do that—at least, this is a very common way that people who use pandas will do to kind of search through that index. The first one is called loc, and there's loc and iloc. That stands for location or integer location. Let's look at loc first. Let's say df.loc, and then we'll do a bracket. Now we're able to specify the actual string, the label. So let's go right up here, and let's say Albania. So we'll say Albania. So again, this is just looking at the location. Let's run this. Now it's going to bring up all the Albania data, just like here where it's kind of looks like a column in a column. And we can get this exact same data but using iloc right here. And when we ran loc, we were searching based off Albania, which is in the 0, 1 position. So if we actually pull the one position for that integer, the iloc, we can look at the one position, and this should give us the exact same data.

Now let's take a look at multi-indexing, and we'll come back to a little bit of this in a second. So multi-indexing is creating multiple indexes. We're not just going to create the country as the index now; we're going to add an additional index on top of that. So let's pull up our data frame. Right now we have the country, but let's do dot reset index, and we'll say in place equals true. Oops, let's run it. So now we have our data frame. Now let's set our index, but this time when we set our index, we're going to add the country as the index as well as the continent as an index. So we'll say data frame.set_index, then we'll do a parenthesis, and instead of just doing country like we did before, we're going to create a list, oops, and we'll do it like that, and then we'll say oops, continent and separate by a comma. So we have continent and country. Let's just say in place is equal to true. Now, when we run this, we're going to have two indexes, and let's see what this looks like. And let's run this. So now we have country as well as continent as our index.

Now you may notice that these indexes are repeating themselves on this continent index. We have Europe right here and Europe right here, as well as Asia and Asia, and it looks a little bit funky. But we are able to sort these values and make it look a lot better. So let's go ahead and try this. We'll do DF dot sort_index, and when we run this, it should sort our index alphabetically. And we can also look in here and see what kind of things we can, you know, specify. We can specify the axis, but it's automatically going to be looking at the zero—this is zero, and this is one. So we have two axes within our data frame. You choose the level, whether it's ascending or not ascending, in place, kind, string, sort remaining—all of these different things. The only one that I really, you know, think is worth looking at is the ascending. I already know some of these other ones. But if we look at ascending, let's run it. Now it's sorted these, and so now it's kind of grouped together, so we have Africa and all the African ones, as well as South America and all the South American ones.

Let's really quickly say pd.set_option, and we'll say display.max_columns, and just like this, let's run it. And I need to specify, whoops, specify right here. Let's see how many rows we have: 235. So let's do 235. Let's run this, and now when we run this, you can see that Africa is all grouped together, and all the countries are in alphabetical order under it, and then we go all the way down to Asia, and again, just all in alphabetical order. If we wanted to, we could say ascending equals true, and then when we run this—Oh, meant to say false—and then when we run this, it's the exact opposite, so it starts with South America, the last one, and then goes in reverse alphabetical order. We could also say false, make it a list, and do comma true, and just like this, and then it would sort this first column as false and this next column as true. So you can really customize it, but, you know, for what we're doing, we don't need any of that; we just need to be able to see this right here.

So now, when we try to search by our index like we did before, we did data frame.loc. Now, when we did that and we said, you know, let's say Angola, when we specified Angola, it's not going to work properly because it's searching in this first index for the first string that we have. We can search Africa. Let's search for Africa, and now we have all of the African countries. And if we want to specify to Angola, we can also go down another level, oops, by doing Angola Angola, and now we have what we were looking at before, where we're calling all the data within those. But we couldn't do it just based off Africa because we had an additional index right here. So once we called both indexes, now we get this view. But let's look at that iloc really quick. When we run this, let's just say one because right up here—oh, we have Angola 0 and then one. So you think it may pull up Angola. Let's go ahead and run this, and it's still pulling up Albania. Let's go right up here. If you remember when we didn't have the multiple indexes, it was pulling up Albania. The difference when you're doing these multi-indexes is that the loc is able to specify this, whereas this one does not go based off that multi-indexing; it's going to go based off the initial index or the integer-based index.

So that's a lot about indexing in pandas. We'll cover even a few more things in future videos as we get more and more into pandas, but this is a lot of what indexing looks like within pandas, and again, super important to learn how to do and know how to do because it's a pretty important building block as we go through this Panda Series. So I hope you enjoyed this video on indexing. If you did, be sure to like and subscribe below, and I will see you in the next video. [Music]

Hello everybody. Today we're going to be taking a look at the groupby function and aggregating within pandas. Groupby is going to group together the values in a column and display them all on the same row, and this allows you to perform aggregate functions on those groupings. So let's start reading in our data and take a look. So we're going to do import pandas as pd, and then we're going to say our data frame is equal to, and we'll say pd.read_csv, we'll do an open parenthesis, R, and our file path, and we're going to be looking at the flavors.csv right here. So right here we have our flavor of ice cream, we have our base flavor—whether it was vanilla or chocolate—whether I liked it or not, the flavor rating, texture rating, and its overall or its total rating. Now these are all my own personal scores, so, you know, I've spent years researching this, so these are all very accurate. But this should be a low-stress environment to learn groupby and the aggregate functions.

So the first thing that we can do is look at our groupby. Now you can't group by—well, you can; you can group by flavor—but as you can see, these are all unique values. What we need is something that has duplicate values or or similar values on different rows that I'll group together. So this base flavor is actually a perfect one to group it on, and we'll do that by saying DF dot groupby, do an open parenthesis, and we'll just specify base flavor, and this will then group together those values. And I need to make sure I can spell properly. This will group those flavors together. So let's run this, and as you can see, it actually is its own object, so it has a groupby data frame GroupBy object. So now that we've grouped them, let's give it a variable. So we'll say group_by_frame. Let's say that's equal to. Let's copy this. We'll run it. And now what we need to do is run our aggregations in order to get an output. So we're going to say dot mean, and that's all we're going to put just for now, just to get an output that we can take a look off, and then we'll build from there. So let's go ahead and run this, and right here we have our base flavor, which is now saying is the index of chocolate or vanilla, and then it's taking the mean or the average of all the columns that have integers. Notice that it did not take the liked column and it did not take the flavor column because those are strings, and they cannot aggregate those, and we'll take a look at that later. But it took all the values that have integers, and then it gave us the average of those ratings.

So right off the bat, as averages with chocolate, I have a much higher rating overall than the ones with vanilla bases. Now we can actually combine all of this together into one line, and we can do something like this. So we'll say DF dot groupby and we'll say mean just like this, and this will actually run. Before, we didn't have any aggregating function on there, so it didn't run, but now that we combine it all into one, it will run properly. Now there are a lot of different aggregate functions, but I'm going to show you some of the most popular ones or the most common ones that you will see. So let's copy this right here. So we can do dot count, and when we run this, we can look at the count, and this will show us the actual count of the rows that were aggregated. So for chocolate, we had three, so there are going to be three all the way across, and for vanilla, we had six, so we're looking at a higher count of vanilla, which, if you're comparing it to this mean up here, that could be a big skew towards the chocolate because if you have one or two good chocolates, it could really pull the numbers up, whereas if you had two good vanillas, but the all the other ones were bad, it pulls that average down. So knowing the count of something is really good. Let's take a look at the next one, and we can do min and max, and I'll just run these really quickly. We can do min, and when we run this, the first thing that you should notice is that it now has a flavor and a liked column, and that's because min and max will actually look at the first letter in the string or the first set of letters. If there are, you know, chocolate something, it'll look at the first, and then it'll actually populate it. So chocolate, with the CH, chocolate is the very first or the minimum value for that string, and for cake batter, that is the minimum value in vanilla as well. Now, with the liked, it's interesting because apparently I liked all the chocolate ones. I'm gonna go take a look. So chocolate, I like chocolate, I like chocolate, I like. So there is no no option in this liked column, so yes was the only option. And now let's look at max, whoops, and it should do the exact opposite, which is going to take the highest value, even if it's a string. So Rocky Road, the letter R comes later in the alphabet, so that's what it's looking at, and so does vanilla, and then we have yes as well, and then, of course, right here it's taking the max value. So before, when we were looking at min, I just focused on those, but it still does the exact same thing to these integer um columns as well. So for the max value for vanilla, it was mint chocolate chip that was our base, so I had a rating of 10 for this vanilla row or grouping. And then we can also look at the sum, and there are all the sums for these, and again, it only does integer because we can't add the strings. Here are the sum or the total values for all of them, and for the total values, since we had, you know, six rows that were grouping into this vanilla, we now have a lot or much higher score for vanilla.

Now that's a really simple way to do your aggregations, but there is actually an aggregation function, and let's take a look at this cuz this is um a little bit more complex, although when I write it out or show you, hopefully it makes a lot of sense. We can do a—so this is our aggregate function—and what we need to pass into our aggregate function is actually a dictionary. So let's do an open parenthesis, and we're going to do a squiggly bracket, and then we need to specify what we're going to be aggregating on or what column. So let's do this flavor rating. Let's copy this. We'll do flavor rating, and I need to put that as a string, and then we'll do a colon, and now we can specify what aggregate functions we want. So we've done sum, count, mean, min, and max—all of those—and we can actually put all of those into here and perform all of those aggregations on just one column. So let's make a list, and then let's say mean, max, count, and uh, what's another one? Sum. So let's do all four of those only on this flavor rating column. And when we run this, we have our base flavor right here, chocolate and vanilla, but now we don't have multiple columns; we have one column with multiple columns of our aggregations. And it is possible to pass in multiple columns like that. So we'll do texture rating, and we'll just come right here and do a comma, then we'll say uh uh texture rating, and then a colon. I don't know why I spelled it out when I copied it, but I did. And then we'll do the exact same ones. And now when we run it, we're getting the exact same columns, mean, max, count, and sum for flavor rating, then mean, max, count, and sum for our texture rating.

Now, so far we've only grouped on one column, but we can actually group on multiple columns. Let's go back up here to our data, and I should have just copied this down here. Let's go back down and just look at this. So really, we only grouped it on this base flavor, but you can do multiple groupings or group by multiple columns. So let's do our base flavor, which we did already, as well as the liked column. So we're going to say DF dot groupby, then we'll do an open parenthesis, and then instead of just passing through one string, we're going to do a list, and we'll say base flavor, oops, comma, and then we'll do liked. So now when it groups this, it should put two groupings. And let's run this and just see. Oops, I gotta say—let's just do mean. So now we have our chocolate and vanilla, and remember, chocolate only had yes, so that's the only one that it's going to group on, but vanilla had a no and a yes. So if we look at the vanilla, we have our base flavor vanilla, and then within liked, we have no and a yes, which can show us that within our vanilla, when we group on these, our nos were really low, but our yeses were really high. We actually had a pretty similar rating or very close to the same rating as the ones we really liked in chocolate. And just like we did above, we can take take this doag, and I'm going to copy this, and it'll perform it on each of those rows. Let me close that. And what did I do wrong? Oh, I need the squiggly bracket, and it'll show us each of those. So the mean, max, count, and sum for all of the chocolate and vanilla, as well as the groupings of liked yes and no.

Now, after we've looked at all that, and that's how I usually do it, there is one uh shortcut function that can give you some of these things just really, really quickly. And so let's go back up here and take this. It's just called describe, um, and if you've ever done it, it's just going to give you some high-level overview of some of those different aggregations. So let's run this, and it's going to give us our chocolate and vanilla, and within each column, it's going to give us our count, our mean, our standard deviation—I believe is what that is—our minimum, 25%, 50, 75, and 100, which is our max, then our count and our mean. So a lot of those aggregate function functions, but the describe is, you know, a very generalized um function. We can't get as specific as we were with the previous ones that we were looking at, but I just wanted to throw this out there in case this is something that you'd be interested in because it, you know, technically is showing a lot of those aggregate functions, just, you know, all at one time. So that is our groupby and aggregate functions within pandas. I hope that that was helpful. I hope that you understood, you know, everything that we were working on. If you like this video, be sure to like and subscribe and check out all my other videos on Python as well as pandas, and I will see you in the next [Music] video.

Hello everybody. Today we're going to be talking about merging, joining, and concatenating data frames in pandas. This whole video is basically around being able to combine two separate data frames together into one data frame. These are really important to understand when we're actually using the merge and the join. Right here we have what's called an inner join, and the shaded part is what's going to be returned; it's only the things that are in both the left and the right data frames. Then we have an outer join or a full outer join, and this will take all the data from the left data frame and the right data frame, and everything that is similar. So basically, it just takes everything. We also have a left join, which is going to take everything from the left and

Then, if there's anything that's similar, it'll also include that. And then the exact opposite of that is the right join, which is going to give us everything from the right data frame, and it's going to give us everything that is similar, but it's not going to give us anything that is just unique to the left data frame. So this is just for reference because, in a little bit, when we start merging these, these become very important. So I just wanted to kind of show you how that works visually.

Let's get started by pulling in our files. So first, we're going to say `import pandas as pd`. We'll run this, and then we'll say `data frame one`, and we'll also have a `data frame two`. These are the different data frames, the left and the right data frame that we'll be using to join, merge, and concatenate. So we'll say `data frame 1 is equal to pd.read_csv`, and we'll do `R`, and here is our file path. So we have this `lr.csv` that's our Lord of the Rings CSV, and let's call that really quickly so we can see what's in there. I'm having a dyslexic moment, uh, because it's supposed to be `reor.csv`. Uh, I apologize for that, but this is our data frame. This is our data frame one. We have three columns: it's their Fellowship ID (1, 2, 3, and 4), their first name (Froto, Samwise, Gandalf, and Pippen), and their skills (hide and gardening, spells, and fireworks). So this is our very first data frame that we're going to be working with.

Let's go down a little bit. Let's pull this down here, and we're just going to say `data Frame 2`. `Data Frame 2`, and this is the Lord of the Rings 2. So let's pull this one in now. As you can see, it's very similar. We have Fellowship ID (1, 2, 6, 7, 8). So we have three different IDs here. We don't have 6, 7, and 8 in this upper, this first data frame. We also have the first name, so Froto and Sam or Samwise are in the very first and the second data frame, but now we have three new people (Barir, Eland, and Legalis), and now we have this age column, which again is unique to just this second data frame.

Really quickly, I want to give a huge shout out to the sponsor of this video, and that is Zenes. I've been using Zenes for my company's customer analytics, and it has been absolutely phenomenal. They're going to be hosting a conference called Zenes Relate on May 10th, and they're going to talk all about customer analytics, chatbots, and AI in this space. You can attend in person in San Francisco, or you can attend virtually, but space is limited, so be sure to apply if you want to attend. So if you are a business leader and you want to make the most out of your customer data, or you want to learn customer data analytics, I believe links in the description. Again, huge shout out to Zenes for sponsoring this video.

Now, the first one that I want to look at is `merge`, and I want to look at `merge` first because I think this one is the most important. I use this one more than any of the ones that we're going to talk about today. The `merge` is just like the joins that we were just looking at: the outer, the inner, the left, and the right. And there's also one called `cross`, and I'll show you that one, although if I'm being honest, I don't really use that one that much, but it's worth showing just in case you come into a scenario where you do want to do that.

So let's go right down here, and I want to be able to see these while we do it. So we're going to say `data frame one`, and when we specify `data frame one` as the very first data frame, we say `data frame.merge`. This is automatically going to be our left data frame. Then, if we do our parenthesis right here and we say `data Frame 2`, this is our right data frame, and let's see what happens when we do this. So what it's going to do, and this we didn't specify this, it's just a default, it's going to do an inner join. So it's only going to give us an output where specific values or keys are the same. Now you can't see this, but what is happening is is it's taking this Fellowship ID and saying I have 1, 1 here, a 2 here. This is the exact same as up here with this Fellowship ID and Fellowship ID of 1 and 2, but when we look at 3 and 4, those aren't in this right data frame, and 6, 7, 8 is not in this left data frame. So the only ones that match are this 1 and 2, and that's why they get pulled in down here. But because we didn't explicitly say here's what I want to join or merge between these two data frames, it actually is looking at the Fellowship ID and the first name. So it's taking in these unique values of Froto and Samwise, which are the same in both, which is why it pulled it over. But really quickly, let's just check and make sure that we did it on the inner join because, again, we didn't specify anything; that was just the default. So we're going to say `how is equal to`, and then we'll say `inner`. If we run this, it's going to be the exact same because, again, the inner is the default. But now, just to show you how it's kind of joining these two uh data frames together, I'm going to say `on is equal to`, and then I'm only going to put `Fellowship ID`. So let's run this.

The first thing that you may have noticed is this `first name_x` and this `first name_y`. What the `merge` does as kind of a default is when you are only joining on a Fellowship ID, we have this right data frame with Fellowship ID, the left data frame with the Fellowship ID. If you're just joining on these and you're not joining on the first name and the first name, then it's going to separate those into an `_x` and an `_y`. And even though they have the exact same values, since we are not merging on that column, it automatically separates that into two separate columns, so we can see the values within each of those columns. If we went into this `on` and we make a list and let's do it like that and we say comma and then we write `first name`, oops, `first name`, and then we run this, it's going to look exactly like it did before. Again, it automatically pulled in both of these columns when it was merging at the first time, even though we didn't write anything. But if we actually write this, it's doing exactly what it was doing when we just had `df2`; we're just now writing it out.

Now there are other arguments that we can pass into this `merge` function. Let's hit shift tab and let's scroll down here. So within this `merge` function, we have a lot of different arguments that you can pass into it. First, we have this `right`, which is the right data frame, which is this `data frame two`. Then we have the `how` and the `on`, which we've already shown how to do. There's a `left_on`, `right_on`, `left_index`, `right_index`; not something you'll probably use that much, but you definitely can if you want to look into that. And there's all these doc strings which show you exactly how to use all of these. So if you're interested in looking at the left and the right and the left and index, it's all in here. The one that is really good is the `sort`, and you can sort it saying either it's false or true. Then we have these `suffixes`. Now, if you remember when we took these out, what it automatically did was it put in these `_x` and `_y`. You can customize that, and you can put in whatever you'd like instead of the `_x` and `_y`. You can put in some custom um string for that. We also have an `indicator` and a `validate`. Again, all things that you can go in here and look at. I'm just going to show you the stuff that I use the most. So these things right here are things that I definitely use the most.

So now that we've looked at the inner join, let's copy this right down here and let's look at the outer join, and these get a little bit more tricky. I think the inner join is probably the easiest one to understand. Let's look at the outer; it's spelled `o u t e r`. I don't know why I always want to say `o t t r`, but let's run this and see what we get. So now this looks quite different. The inner join only gave us the values that are the exact same. This one is going to give us all of the values regardless of if they are the same. So we have 1, 2, 3, 4, 6, 7, even and 8. So let's scroll back up here. So we have 1, 2, 3, 4, 1, 2, and 6, 7, and 8. So we don't have a 5, and then if you notice in this data frame right here, if the value doesn't have, so if we can't join on the Fellowship ID or the first name, like Legalis wasn't one that we joined on or that has a similar value in the left data frame, it just gives us an `NaN`, which is not a number, and it's going to do that for any value where it couldn't find that join or it couldn't match uh something within that either ID or first name. So in age, we also have that for the ones that weren't in the right data frame. We only had 1 and 2, so we'll have the age for both Froto and Sam, but for Gandalf and Pippen, we don't have their corresponding IDs, and so it's just going to be blank for Gandalf and Pippen, and you can see that right here. So again, outer joins are kind of the opposite of inner joins; they're going to return everything from both. If there is overlapping data, it won't be duplicated.

Now let's go on to the left join, and I'm going to pull this down right here, and now we're just going to say `how is equal to left`, and let's run this. So what this is going to do is it's going to take everything from the left table or the left data frame right here, so everything from `data frame one`. Then, if there is any overlap, it'll also pull the overlapped or the, you know, whatever we're able to merge on from `data Frame 2`. So let's go back up to our `data frame one` and two. So it's going to pull everything from this left data frame because we're specifying we're doing a left join, so everything from the left data frame will be in there. We're also going to try to bring in everything from the right, but only if it matches or or is able to merge. So just this information right here will come over. We weren't able to join on 6, 7, or 8, so really none of that information is going to come over. So let's go down and check on this. So again, we have 1, 2, 3, 4; all of the data with this first name and skills, everything is in here, but then we are trying to bring over the age, but we only have matches with 1 and 2, so only these two values will will come in.

Let's look at the right join. See, it's basically the exact opposite. Let's look at the right, and this is basically the exact opposite of the left, in the fact that now we're only looking at the right hand, and then if there's something that matches in `data frame one`, then we will pull that in. So this is basically just looking like `data Frame 2`, except we're pulling in that skills column, and since only 1 and 2 are the same, that's why this skills values are here.

Now those are the main types of merges that I will use when I'm using a data frame or when I'm trying to merge a data frame, but there also is one called a cross or a cross join, uh, and let's look at this one, and this one is quite a bit different. Here we go. Let's run this. So this one is different in that it takes each value from the left data frame and compares it to each value in the right data frame. So for Froto in this left data frame, it looks at the Froto in the right data frame, Samwise in the right data frame, Legalis, Elron, and Baromir, all in the right data frame. Then it goes to the next value; Samwise does the exact same thing: Froto, Samwise, Legalis, Elon, Baromir, and it does that for every single value. So let's go right back up here. So it's taking this, this 1, it's comparing it to 1, 2, 3, 4, 5. Then it's taking Samwise; it's comparing it to 1, 2, 3, 4, 5. Gandalf, 1, 2, 3, 4, 5. Pippin, and then you kind of see that pattern, and that's what a cross join is. Um, there are very few, in my opinion, reasons for a cross join, although you'll, if you ever do like an interview where you're being interviewed on Python, you will sometimes be asked on cross joins, but there aren't a lot of instances in actual work where you really use or need a cross join.

Now let's take a look at joins, and joins are pretty similar to the merge function, and it can do a lot of the same thing, except, in my opinion, the join function isn't as easily understood as the merge function. It's a little bit more complicated, um, but let's take a look and see how we can join together these data frames using the join function. So let's go right up here. We're going to say `data frame one.join`, and then we'll do `data frame two`, very similar to how we did it before, and let's try running this, and it's not going to work. Um, when we did the merge function, it had a lot of defaults for us. Let's go down and see what this error is. It says the columns overlap but no suffix was specified. So it's telling us that it's trying to use the Fellowship ID and the first name just like the join did, except it's not able to distinguish which is which, and so we need to go in there and kind of help it out a little bit. Again, a little bit more hands-on than the merge, but let's see what we can do to make this work. Let's do comma, and we'll say `on`, and let's really quickly, let's open this up and kind of see what we have. So this one has, has less options than the merge does. We have `other`, and that's our other data frame. We can do `on`, and we're going to specify, you know, what column do we want to join on, and then we can look at how do we want it to be: a left, an inner, and outer, the same kind of types of joins as the merge. Then we have that `lsuffix`, `rsuffix`, and that's right here is kind of part of the issue that we were just facing is that those columns are the same, but if we say `lsuffix`, it'll give us an underscore whatever we want to specify any string for columns that are both in the left and the right; we can give it a unique name, so it will no longer have that issue, and then we can also sort it like we did on the other one. But anyways, let's go back to our `on`. We'll say `on is equal to`, and then we'll say `Fellowship ID`. Let's try running this, and we're still getting an error; it's just not as simple as the merge. So let's keep going. So now let's specify the type, so we'll say `how is equal to`, and we'll do an `outer`, and if we run this, it still doesn't work; we're still getting the exact same issue as the `lsuffix` and the `rsuffix`. So now let's finally resolve it. I just wanted to show you how a little bit more frustrating it was, but now let's say uh `lsuffix is equal to`, and now it automatically, when we did the merge, did an `_x`, but we can do let's do `_left`, and then we can do a comma, we'll do `rsuffix`, and we'll say `is equal to`, and we'll do `_right`. Now when we run this, it should work properly. Let's run this. So this is our output, and obviously looks quite a bit different over here. We have this Fellowship ID, then we also have `Fellowship ID_left`, `first name_left`, `Fellowship ID_right`, and `first name_right`. So it just doesn't look right. Now something I didn't specify when I first started this because I kind of wanted to show you is that the join usually is better for when you're working with indexes. Before, when we were using the merge, we were using the column names, and that worked really well, and it was pretty easy to do, but as you can see right here, when we're trying to use these column names, it's not working exceptionally well. Let's go ahead and create our index, and then I can show you how this actually works and how it works a little bit better when we're working with just the index, although you can get it to work just the same as the merge, it's just a lot more work.

So let's go right down here, and let's go and say `df4`. So we'll create a new data frame. We'll say `df1.set_index`, and we'll do an open parenthesis, and we'll say we want to do this index on the Fellowship ID, and then we're going to do the join. So now we're going to say `join`. So we're setting an index, so we're setting that index on the Fellowship ID. Now we're going to join it on `df2.set_index`, and then we're also going to do that on the Fellowship ID, and I'll just copy this. Oh geez, I hate it when I do that. Okay. Now we also want to do and specify the left and the right index, so I'll just copy this, as we do need to specify this. Now let's try running the `data frame 4`. So really quickly, just to recap, we were setting the indexes; we were doing the same thing above, right? We have this join; we were joining `data frame 1` with `data Frame 2`. Now we're joining `data frame 1` with `data frame 2`, except in both instances we're setting the index as Fellowship ID, so we're joining now on that index. So now let's run this, and this should look a lot more similar to the merge than the join that we did above, except now the Fellowship ID right here is actually an index, so it's just a little bit different, but we can still go in here and do `how is equal to Outer`. Oops, let's say `outer`. So we can still specify our different types of joins or the different way that we can merge or join these data frames together. We can still specify that. Again, it's just a little bit different, and that's why, for most instances, I'm using that merge function because it's just a little bit more seamless, a little bit more intuitive. The join function can still get the job done, but as you can see, it takes a little bit more work.

Now let's look at `concatenate`. Concatenating data frames can be really useful, and the distinction between a merge and join versus the concatenate is that the concatenate is kind of like putting one data frame on top of the other rather than putting one data frame next to one another, which is like the merge and the join. So concatenating them is just a little bit different in how it'll operate, but let's actually write this out and see how this looks. Let's go up here, and we'll say `pd.concat`. We'll do an open parenthesis, and then we're going to concatenate `data frame one`, comma, `data Frame 2`. That's all we have to write, and let's run this. And so, just like I said, it literally took the first data frame (1, 2, 3, 4) for and put it on top of the right data frame (1, 2, 6, 7, 8). So that is our left data frame; this is our right data frame, and they're literally just sitting one on top of the other. But just like when we merge, either with a left or a right, when you have these skills and there aren't any values that populate for them, it is going to say not a number. And since we're not actually joining, we're not joining on 1 and 2, even though this 1 and this 1 is the same rows, it's not populating that value because, again, we're not joining these together; we're just concatenating and putting one on top of the other. Now if we go into this `concat`, we say shift tab; there are a lot of different things that we can do, which, if you remember, the zero axis is the left and index and the axis of one is the top index, which is the columns, so you can specify that, and we can also do joins and

this is the one that I'm going to take a look at, but there are other ones that you can um look into as well. Let's look at join. Let's do comma, and we'll say join is equal to, and let's do an inner join. So let's see what happens with this. As you can see, it is only taking the columns that are the same; that's what this inner is doing. It's joining these columns together, and the ones that were different, they didn't take because, again, we weren't able to combine them; they aren't similar between both data frames.

Let's do an outer, and now it's going to take all of them. And like I said, that's doing this on these columns right here, but we can also do it on this axis as well. So, so let's go ahead and say axis is equal to 1, and when we run this, now it's joining us on this Index right here of 0, 1, 2, 3, 4. So now these ones are being joined together, and it's putting it side by side, much like a merge would. So that's how concatenate works. And I'm going to show you one more thing, and again, it's not up here in this, you know, title because it's not one that I recommend, but it's one called append. The append function is used to append rows from one data frame to the end of another data frame, and then we can return that new data frame. And so let's do dataFrame1.append; we'll do an open parenthesis, and we'll say dataFrame2. Very similar to how we've been doing other things, and let's run this. And as you can see, this is almost exactly like how the concatenate did when we first did it, but if we read kind of this warning, it's saying the frame.append method is deprecated and will be removed from pandas in the future version; use pandas.concat instead. So it's literally warning us, you know, append is on its way out. If you want to do exactly what you're doing right here, go and try concat or concatenate because that'll do the exact same thing. So I'm not really going to show you any other variations of append because there's no reason; it's going to be on its way out in the next version. So that is our video on merge, join, and concatenate, and append as well, uh, in pandas, and I hope that that was helpful. I hope that you learned something. I mean, this stuff is really important because often times you're not just working with one CSV or one JSON or one text file; you're working with multiple of them, and you you need to combine them all into one data frame. And so this is a really, really important concept and thing to understand. With that being said, be sure to like and subscribe, check out all my other videos on Python and pandas, and I will see you in the next [Music] video.

Hello everybody. Today we're going to be building visualizations in pandas. In this video, we'll at how we can build visualizations like line plots, Scatter Plots, bar charts, histograms, and more. I'll also show you some of the ways that you can customize these visualizations to make them just a little bit better. With that being said, let's go right over here, start importing our libraries, and we'll start with importing pandas as pd. And this one is really all you need to actually create the visualizations in pandas, but we may get a little bit crazy, uh, and so we're going to do a few different ones as well, like import numpy as np, and then we're going to do import matplotlib.pyplot as plt. Now I may or may not use this; I just, you know, when I get into visualizations, I may want to change some different things, so we're going to at least have them here in case we do want to use them. Let's go ahead and run this.

So now let's get our data set that we're going to be using. So let's say dataFrame is equal to pd.read_csv, and let's get this in right here. Now we're going to be doing these ice cream ratings. Let's take a look at this really quickly. Now these values are completely randomly generated; they are not real in any way, um, but that's what we're going to be using. So I just wanted something kind of generic, something that wouldn't be too crazy confusing, just something that we could use and you guys can understand that they're just numerical values. But let's also set that index really quick, so we'll say dataFrame.set_index, and then we'll say date, and then we'll say that that's equal to the dataFrame. And we have this date column right here as our index. So we have uh January 1st, 2nd, 3rd, 4th, and then we have our ratings right here, and again, these are all just integers, and they're pretty easy, or really easy, to demonstrate how you can visualize these, so that's why we're using it today.

So the way that we visualize something in pandas is we use something called plot. So let's just take our dataFrame; we'll do a dataFrame.plot, and we'll do our parentheses. Now let's go in here really quickly; let's hit Shift+Tab, and this is going to come up, and this is pretty important because this kind of is going to tell us what we can do within this plot. And unfortunately, there isn't like a quick overview; we just have this docstring, but we have our parameters right here; these are what we can pass in to kind of customize our visualization. So the data is going to be our dataFrame, then we have our X and Y labels; we can specify the kind, and this one's important because we can specify what kind of visualization do we want. We can do a line plot, horizontal, a vertical bar plot, histogram, box plot, and then a few others, including area, Pi, density, all these other things. We can also specify if we want it to be a subplot, and a lot of these things that I'm specifying, you know, I'm going to show you how to do; you can use uh different indexes; you can add titles, add grids, Legends, Styles, all these different things. I mean, you can go through here because there are a lot, but you can specify and, you know, customize all of these things. We won't be going into all of them, but I will show you some of the ones that I probably use the most and that I think are the most useful to know right away. So let's get out of here, and we're just going to do DF.plot, and when we run this, we'll get this right here, and that was super, super easy; created a line plot by literally doing just about nothing, um, but by default it's going to give us a line plot. So if we come up here, we say kind, and let me get that out of the way, is equal to 'line', and we run this. So by default, without us actually having to input anything, it's giving us that line plot as a default. So uh we can specify it's a line plot. As you can see, we already have all of our data right here; we didn't have to specify anything; it kind of automatically took it in; it is visualizing all three of these columns, and it has this little um Legend right here, and we can specify where we want that; there is an argument to be able to do that. It also gave us these tick marks of 2, 4, 6, 8, 10. Again, it read in and it said it's only going from 0.0 to 1.0; that is kind of the peak, and so it kind of automatically gave us these ticks for us. Again, that's another thing that you can specify; we make it go up to 2, 5, 10, 1,000, whatever you want it to be, and then we're doing this based off of this date value right here.

Really quickly, I wanted to give a huge shout out to the sponsor of this entire Panda series, and that is Udemy. Udemy has some of the best courses at the best prices, and it is no exception when it comes to pandas courses. If you want to master pandas, this is the course that I would recommend; it's going to teach you just about everything you need to know about pandas. So huge shout out to Udemy for sponsoring this Panda series, and let's get back to the video. If we wanted to break these out by the actual column, we could go in here and say subplots is equal to true, and it's actually subplots, whoops, and now we can run that, and then we can see each of those columns being broken out by themselves instead of them all being in one visualization; that's now uh three separate visualizations.

Now let's go right over here; we're going to get rid of the subplots. I want to show you just some of the different arguments that you can use to make this look nice, uh, because I don't want to do this on every single visualization; I just want to show you what you can do. So we have this one right here; we can add a title; notice there's no title or anything really telling us what that is, so we can say comma title, and we'll say 'Ice Cream Ratings'. If we run this, we now have this nice title right here. Now we can also customize the labels or the titles for the X and Y axes. It automatically took this date, which is right here; this is our date index; it automatically took that for us, but we can customize that if we'd like to. All we have to do is comma, and then we'll say xlabel is equal to, and so our X is this date one right here, and we can say 'Daily Rating', and then we can do the ylabel; we'll say ylabel is equal to, and for this one we can say 'Scores'. Hope you cannot hear my dog in the background cuz they're being insane, uh, but let's go ahead and run this, and now we have these daily ratings on the x-axis, and on the y-axis we have scores.

Now let's go right down here and start taking a look at our next kind of visualization, which is going to be a bar plot. So we'll do DF.plot; we'll do kind is equal to, and for this one we're going to say 'bar'. Now this is what your typical bar plot will look like, and a lot of the arguments that we just did on the line plot you can also apply to this bar plot. Something that's unique to the bar plot is that you can also make it a stacked bar plot. All we have to do is go in here; we'll say comma, and we'll say stacked is equal to true. So now this is going to make it a stacked bar chart instead of just, you know, your regular bar chart. Let's go ahead and run this, and as you can see, this is now stacked on top of one another, with each of these columns all representing the values that they have. Now we don't always have to to do every single column; we can also specify the column that we want. So let's take the flavor rating, for example; we could do flavor, oops, flavor rating, good night, flavor rating, and then it's only going to take in that flavor rating column, and if you notice, we don't have a legend; that's only when you have multiple values, which we are only looking at this one column, so all the values are right here. Now in this bar chart, it automatically defaults to a vertical bar chart, but you can change it to a horizontal bar chart. Let's go ahead and take a look at how to do that. Bring back all of them; we'll do DF.plot.barh, and I don't know if I can keep in that kind equals bar; let me run this. Yeah, I need to get rid of that because the barh is its own um this is its own function. So now I'm going to run this; it should just have a stacked bar chart, except now it should be horizontal. So now you can see this worked properly; it's basically the exact same thing as a vertical bar chart, just now horizontal, which may look better, especially depending on if you have values like this or, you know, something else that just looks better being horizontal.

Now the next one that we're going to take a look at is the scatter plot. So we're going to say DF.plot.scatter, and if we run this, we're going to get an error. What we need in order to run this properly is we need to specify the X and the Y axes in order for this scatter plot to work. So let's go here, and we'll say x is equal to, and we can take any of our columns that we have up here, so we'll say x is equal to 'texture rating', and then oops, y is equal to, we'll do 'overall rating'. Now when we run this, it should work properly. Let's go ahead and take a look. Now if we go in here and we do Shift+Tab, we can also see some other things that we can specify. So let's go right down here. So we have our X, and we have our Y, and those are the ones that we just did; we can also pass through an S, which is going to tell us or or change the size of the actual dots right here in our scatter plot, then we can also do a c, which is the color of each point. Let's start with the S; let's say s is equal to, and let's just do 100; let's see what that looks like. So we have a much larger number; let's do 500 and see what that looks like. So we can make these much larger on our visualization, depending on what you're looking for. We can also look at the color; let's put comma c. So for color, we can say color is equal to, and let's do uh yellow; let's see if this works. So now we've changed it to yellow; that looks absolutely terrible, but it does work.

Now let's move on to the histogram. Histogram is always a good one; it's very similar to something like a bar chart, but what's great about a histogram is you can specify the bins, um, so let's go ahead and say DF.plot.hist, then we'll do an open parenthesis, and let's go ahead and hit Shift+Tab in here; take a look at this one as well. So some of our parameters are the actual columns or the data frames that we want to pull in; we can choose the bins, and they have a default of 10 in here, and so let's take a look at how this works. So we'll just run this as it is. So this is by default what this histogram is going to look like. Let's go ahead and specify our bins; we'll just say it was 10 by default; let's just do 20, see what that looks like. So there are smaller columns right off the bat, and remember histograms are really good for showing distribution of variables, you know, that's really what a histogram is for, but of course since these are completely random numbers, this histogram isn't going to make any sense at all, but you can at least kind of see visually how it works. And if I didn't mention it before, which I should have, the bins represent how many kind of tick marks are down here. So if we just do one, it's only going to be one very large uh, you know, histogram. We could even go further down from 10 and do 5, so now there's only one, two, three, four, five. So the distribution gets smaller, and things get more compact as you spread it out; again, like we did 100, it's going to spread it out a lot, um, and this is what it shows; you know, it's showing the distribution of those bins across however many you want. So the 10 by default, you know, it usually is pretty good for a lot of different things.

Now let's go down here and look at the box plot, and the box plot is a pretty interesting one. Let's go ahead and visualize it really quickly, and then I'll kind of explain how this one works. Let's do DF.boxplot; let's run this, and really what we're looking at is some different markers within our data. This line right here is the minimum value within that column; we also have the bottom of the box, which is the 25th percentile of all the values within just this column; this is 50%; then we have 75%, and then up here we have our maximum value. So I can take a glance at this and see that we have a low minimum, a high maximum, and it definitely skews towards the lower range, whereas if I look over here, we have a lower minimum and a higher maximum, and you can see that this medium point is at 6 versus 0.4 over here, so this skews a lot higher.

Now let's go down here and take a look at an area plot. We'll do DF.plot.area, and let's just run this. This is what we're going to get by default. Now something I wanted to show you earlier; I just haven't gotten around to; I want to show you something called Figure size or figsize, um, so for this, it's, you know, it just looks small; looks a little bit cramped. Let's say we want to increase the size of this, and we'll say figsize, oops, figsize is equal to, and let's just do a parenthesis and say 10, 5. That should be pretty large; this is going to make it a lot larger; just something I wanted to throw in there. I look at these area charts as pretty similar to like a line chart; if we went and compared those, be pretty similar, um, but they're different visually, and, you know, you absolutely can use these for different types of visualizations, but I don't use this one a lot, if I'm being honest; that's why it's kind of towards the end of the video, but you definitely can do it.

Let's go on to our very last one of the video; that's going to be the beautiful pie chart. Let's do DF.plot.pie, open parenthesis, and let's run it. We're going to get this error; that's because we need to specify what column we're working with here. So let's just say the Y, and that's what we need. Let me open this up for us right here; we have our Y, and this is our our label or our column that we're going to plot; that's really all we need. So we can just say y is equal to 'flavor rating', oops, 'flavor rating'. Let's run this. Now we get this visualization right here. Let's make this one a little bit bigger; figsize is equal to 10, 6. Now it's a little bit bigger; it definitely depends. So this Legend is going to auto-populate; you know, you can make this as big as you want, and obviously it's going to look a little bit better if you do it larger, and these colors auto-populate. Now you can customize these colors, although I found these ones to be just when you have a lot of of them; it's harder to customize them as easily, but, you know, definitely look into it. These are things that everything in here is almost something that you can customize in some way, although it does get a little bit tricky; you definitely have to do some research and some Googling around just to kind of figure out how to do those things.

Now one last thing that I wanted to show, and something, you know, I could have probably done at the beginning, um, is you can actually change what visual this is, and we can do that pretty easily within matplotlib. There are different styles, um, and so let's go right here; let's add a new row, a new cell, and we'll say print, and we'll do plt, so that's that matplotlib right here; we'll do plt.style.available, and what this is going to do, whoops, what this is going to do is show us all these different types of stylings that you can do to kind of change up this visualization. Then once we find the one that we like, we'll just do plt.style.use, and then in the parenthesis we'll just specify which one we want. Now there's all these Seaborn ones, and Seaborn is a really great um really great Library. Let's try Seaborn-deep; I haven't tried this one at all; let's go ahead and try this; just changes some of the colors, some of the visuals. We can try something like 538; let's try this; that looks quite a bit different. And let's try something like um classic; I don't know what this one looks like; let's just try it. So you can try out all these different styles; find one that you'd like; find one that you think looks really nice, and you can run with it through all your visualizations. So this has been our video on visualizing data in pandas; I think it's a really good introduction on how you can visualize data within Python, and in future videos we'll look at matplotlib and Seaborn, which are some

Really great libraries for visualizing data, which I use a lot. So I hope that you enjoyed this video. If you did, be sure to check out all my other videos on Python and pandas, and I will see you in the next [Music] video.

Hello everybody. Today we're going to be cleaning data using pandas. Now there are literally hundreds of ways that you can clean data within pandas, but I'm going to show you some of the ones that I use a lot and ones that I think are really good to know when you are cleaning your data sets. So we're going to start by saying import pandas as pd, and we're going to run that. And now we're going to import our file, so we're going to say data frame is equal to pd, that's pandas, dot read_uncore, and we actually have this in an Excel file, so we'll say read—oops—say read_excel, do an open parenthesis, and we'll do R, and then we'll paste the path right here. And now we're just going to call that variable, so we'll call data frame and we'll actually read it in and look at the data. So let's scroll down here and let's let's take a look at this data frame or this Excel file that we're reading in.

So right off the bat, we have this customer ID that goes from 101 all the way down to 20. We have this first name, and everything looks pretty good here except in this last name column; uh, looks like we have some errors. We have some forward slashes, some dots, some null values, um, so definitely going to have to clean that up because we don't want that in the data. We have a phone number, and and it looks like we have a lot of different formats, um, as well as NaNs—not a number—um, just lots of different stuff, so we're going to need to standardize that, so clean it up and then standardize it to where it all looks the same. Um, we also have address, and it looks like on some of these we just have a street address, but on some of the other ones we have like a street address and another location as well as a zip code in some of them, so we'll probably want to split those out. We have a Pay customer, uh, which is yes and Nos, and some of those are not the same, so I have to standardize that. We have a do not contact, kind of the same thing as the paying customer, and we have this not useful column which we'll probably just want to get rid of. Okay, so the scenario is is that we got handed this list of names and we need to clean it up and hand it off to the people who are actually going to make these calls to this customer list. So they want all the data in here standardized and cleaned so that the people who are making those calls can just make those calls as quickly as as possible, but they also don't want columns and rows that aren't useful to them. So things like this not useful column we're probably going to get rid of, and then ones that say do not contact, if it says yes, we should not contact them, we probably will want to get rid of those somehow. So that's a lot of what we're going to be doing to clean this data set.

Normally the very first thing that I do when I'm working with a data set, most of the time—except very rare cases when you're actually supposed to have duplicates—is I actually go and drop the duplicate from the data set completely. All you have to do for that is say df.drop_duplicates. So they make it super easy for you. Let's just run it, and up here is our original data set. We have this 19 and 20, and those are obviously duplicates; they have the exact same data; it's just a duplicate row that we need to get rid of. If we look right down here, we no longer have that 20; we now just have one row of Anakin Skywalker. And of course, of course, we want to save that, so we're just going to say df is equal to and df. So now it's going to save that to the data frame variable again, and now when we run this, our data frame now does not have any duplicates. That's definitely one of the easier steps that we're going to look at; things are going to get quite a bit more complicated as we go, but I'm starting out, you know, kind of simple so that we can kind of get a feel for it, and then we'll start getting into the really tough stuff.

So the next thing that I want to do is remove any column that we don't need. I don't want to clean data that we're not going to use. So if we're just looking through here, you know, they may need, you know, first name, last name, phone number for sure; address might give them some information of where they're calling to or time zone, so we want that. This not useful column looks like a pretty good candidate to delete, and it's very easy to do that. We're going to go right down here, and we're going to say df.drop, we'll do an open parenthesis; drop just means we are dropping that column, and we can specify that by saying columns is equal to, and then we'll paste in that column that we want to delete. So let's run this and see what it looks like, and it literally just drops that column exactly like we were talking about; it no longer has that column. Again, we want to save that; we can always do in_place equals true, um, if you follow this tutorial series, you can always do in_place equals true, and that'll save it as well, but just for our workflow, most of the time I'm going to assign it back to that variable, um, just for keeping it the same.

Now let's kind of go column by column and see what we need to fix, and we'll start on this left-hand side. This customer ID to me looks perfectly fine; I'm not going to mess with it at all. The first name at a glance also looks perfectly fine; I don't see anything wrong with it visually, which is a good thing, um, although sometimes that can be deceiving and that can cause errors down the line, but we're not going to uh assume that there are errors in here. Now let's look at this last name. Now the last name obviously I'm I'm seeing some obvious things, things that we talked about when we were first looking at this data set. We have this forward slash, which we definitely need to get rid of. We have values, so not a number right here. We have some periods as well as an underscore right here, so all those things I think we should clean up and get rid of it so that when the person is making these calls, you know, it's all cleaned up for them. So how are we going to do that? We can actually do this in several different ways, but let's just copy this last name. The first one I'm going to show you is strip, and we'll write it kind of like this: we'll say data frame and then we'll specify the column that we're working with because we don't want to make these changes or strip all of these values from everywhere; we only want to do it on just this column. If we do this and we don't specify the column name, it will apply it to everywhere. So if we're trying to do these, yeah, let's say bum these underscores, maybe that would mess with something else in another column, and we don't want that. So we just want to specify just this last name. So let's go last_name.string.strip. Now what strip does, and let's see if we can open this up really quickly—no, we can't—but what strip does, I was just I was hitting shift tab in here to see if it could bring up, um, you know, some of the notes on it, but what strip does is it takes either the left side or the right side; lstrip takes from the left side, rstrip takes from the right side, and strip takes from both, but you can strip values off the left and the right-hand side, and we can specify those values. Now for what we're doing in this column, we can just use strip because, as you can see, this forward slash, these dots as as well as this um underscore are all on the far sides. If there was a value like swancore Son, the strip wouldn't work at all because it's not on the outside of of the value or the word. So we can use strip. I'll also show you how to use replace, and replace is another really good option for things like this, but let's start with strip and just see what it looks like and see if we can get what we need done. So let's just run this for now, see what happens. So it looks like nothing has changed because, again, we're not specifying any specific value; just by default, it's only taking out white space, so like spaces that shouldn't be there; that's what it does by default. Now we can specify within this exactly what values we want to take out. So let's go ahead and do that. Let's say left strip and let's try to take out these dots real quick. So we're just going to do a parenthesis dot dot dot. Now let's run this and see what it looks like for this one. Water it is now gone. So those three dots were there before; let's just show it, so they were there, and then when I ran it like this, now they're gone. That's what the lstrip does; it takes it only off the left-hand side. Now we can also do a forward slash, so we'll do something like this, and it'll get rid of the white, but as you can see now we aren't taking out these three dots, so they're still there. Now is it possible to do something like this where we put these values inside of a list? Um, let's try it. So we'll say just like this 1 2 3. Let's run it, and no, it doesn't. Um, this lstrip actually sits within the the realm of regular expression, so if you've ever worked with regular expression, you know it gets very complicated, very complex, so you want to keep it kind of simple, especially with these values where we're just taking a few out. So what we're going to do is we're going to do dot dot dot, and we're take it out one by one. Now in order to save this CU, we want to save this; we want to take out that value; we don't just want to say data frame equals because that would be uh very bad. What this would say is now this data frame is only equal to these values that we're seeing right here. We want to only apply it to this column, so we're going to go like this. So now when we do it and then we call the entire data frame, it's only applying this to this one column, the last name column. So let's run it, and now when we go down to Potter right here, it's cleaned up. So we're going to do the same thing but for those other values, and we'll do it just like this: we'll do a forward slash and it's a left strip, and then we'll do I'll do the left strip on this underscore just to show you that it won't work, and then we will go on from there. So it's not pulling it because we're looking at the left-hand side only; we need to use rstrip. So now let's use rstrip, and and now that looks perfect; has no underscore. So that's how you can use strip for either the left side, the right side, or just strip by itself, which covers both sides.

Now I showed you all of that because I am going to show you a different way to do it, um, and I apologize because I somewhat lied to you earlier. Um, let's run this right here. Actually, we're just going to pull it in like this; we're going to remove the duplicates again; bear with me; we're going to drop that column, and then now we're sitting with that data frame again with those exact same mistakes. I just wanted to reset it for a second. There is a way uh that you can do this, and I just wanted to, you know, kind of show you how you can do it. You can do this right here, and we'll say—so we're now again we're just looking at this column, just this column—and we're using strip, and let's get rid of R because we want to do apply it to everywhere; you can input all of those values individually, and it will clean it up. So let's say we want to get rid of numbers; we'll do 1 2 3, then we can do the dot, so that's going to be for our period or for a dot dot dot Potter; we could also do the underscore, and we can do the forward slash. So we put it all in one string right here. Now let's take a look at this; we'll get rid of this really quickly. Now let's take a look, and all of them were removed. I showed you how to do it before because that's at least how my mind would think about it; I'd think, oh, I can put it in a list and run it through this lstrip or this right strip, and it would work, um, but that's not how strip works; you have to kind of combine it all into one value. So uh yes, I deceived you; I apologize, but now when we call data frame and we assign it to that column, so the last name column, or assigning what we just did to this last name column, everything should look perfect, and it does. So our customer ID, first name, last name are all cleaned up.

Now we're going to come to a much more difficult one; this is probably, if I'm being honest, the hardest one. I said we were going to work up, but this is probably the hardest one of the whole video: working with phone numbers. And look at all these different types of of formats; I mean, it is um it's not going to be fun, and imagine, you know, there's 20,000 of these; you can't just go and manually clean those up; you need something to kind of automate that. So that is what we're going to do. So let's go right down here; we'll copy the data frame, and I'm going to pull it right here. So now we need to clean up this phone number. What we want is it all to look exactly the same, unless it's blank, and we'll keep it blank; we don't want to populate that data, but we want all of them to look exactly like this one. And what we're going to do is right off the bat, we're going to take all of the non-numeric values and just completely get rid of them; strip it down to just the numbers. So this 1 23-643 or forward slash will just be the numbers; same with these bars and these slashes and everything; all of these will just be numeric; then we'll go back and reformat it how we want to format it, which will look exactly like this one, um, but we just want to do it for the entire column. So let's go right up here, and we're going to try replace for the first time. So let's do phone_number, just oops, that's not what I wanted. So we're going to do a bracket, say phone_number.string.replace, just just like we did before. Now we're going to use some regular expression in here, and I'll kind of do a really high overview, although I'm not going to dive super deep into the regular expression; then we're going to do a parenthesis, and within there we're going to do a bracket; um, I can't remember what this is called; is it called a carrot? I think it's called a carrot; uh, B; I'm just going to call it that; it may not be correct, but I think it's an upper arrow, so it's an upper arrow, a-h—oops—A-Z, A-Z, and then 0-9. Now at a super high level, what that character, that first thing is doing, it's saying we're going to return any character except, and then we specify anything A to Z, A to Z, upper or lowercase, and then actually I think this should be like this: A to Z, uh, and then 0 to 9. So any value like a b c 1 2 3, those are not going to be matched; it's going to match all of them except these values, and then we're going to replace them by saying comma, and we're going to replace them with nothing, so this is just an EMP empty string. So literally we're taking everything that is not an A B C A 1 2 3, so a letter or a number, we're replacing all of that, and then we're replacing it with nothing. So let's run this and see what it looks like, and it looks like that worked properly. Now we do have this na because we had an n-a for I don't remember, maybe that was Creed Bratton, um, but it worked for basically everything else. We're going to go through the entire process, and then at the end we'll remove any values; we want them to just be completely null; we we don't want them to even see n an and wonder what that is; we just want it to be blank, and we'll do that at the very end. So now that we know that that worked, let's assign it; we'll do df phone_numers equal to, and then we'll say data frame. And this looks a lot more standardized than it did before already, but now what we want to do is try to format this, um, and I've done this many many times; I always use a lambda; you can definitely use a for loop; I just I don't do it that way myself, so I'm going to show you how to do it using a lambda. Let's get rid of this, and we're going to say df phone_number; we've already done that; I'm just going to get rid of it; we're going to say df phone_number, then we're going to say dot apply, we'll do an open parenthesis, and then this is where we're going to build out our lambda, so we'll say lambda x colon. Now this is where we're going to kind of format it. So what I want to do is I want to take the first three string strings 1 2 3, then I want to add a slash, and then the next three strings, add a slash or a dash, uh, and then that be the value that's returned. So it's not super difficult; we're just going to do x then a bracket; let me get rid of that; an x and then a bracket, and then we want the 0 to 3, so it goes 0 1 2, so 0 1 2; it doesn't include the three; it goes up to three, so 0 1 2; that's our first three values; then we'll do plus and do a quote and do a dash; so this is our first kind of sequence, and I'm just going to copy this; we'll do plus, and instead of three, or we are going to start at three because now it's inclusive, so we're going to go from three and we're going to go all the way up to six, so it should be three four five, our next three values, then we have a dash, and we'll copy this, and we'll say plus, and now we go from six all the way to 10. Now let's try running this, and as you can see, we get an error. Now I already know what the error is: float object is not subscriptable, which means we're trying to um basically look at it like a string right now; it's not a string; it's actually a number. So let me get rid of this for just a second; I'm going to show you what it's talking about. So right now we have values that are floats and values that are strings or not even a number. So if we want to actually look through it like kind of like indexing, if we want to do that, they all have to be strings, so we need to change this entire column into strings before we can apply this um formatting. Now when I was creating this, if I'm being honest, my first thought when I was doing this was to do it like this: string df phone_number, um, let's just run that; this is what

The values look like um, and I don't remember why or why it was doing this. I can't, I can't remember, but I looked into it quite a bit and I was like, oh, I need to apply this string, converting it to a string on each value, not the entire row or not the entire column. So how we can do that is actually fairly easy because we've already done a lot of the heavy lifting; we're just going to copy this, and we're going to say x, so string of x. And again, Lambda is like a little anonymous function, so you could do this by saying for um, X in this uh column, we could do a for loop and then say for every X, it equals the string of X, and then it changes it to a string, but a Lambda just does it a lot quicker. Um, so we're going to say, so let's do that really quickly, and all of our values look exactly the same, and that's how we want it. So we're just going to copy this, apply it, good. And now we're going to take this and we're going to run this again; just ignore all my commented-out stuff, pretend I don't have that. Um, so now when we run this, it should work. There we go. Now if we look at these numbers: 1, 2, 3-545, D, 5421, and it does that for every single one where there's values, even when there's Nan or na, it's still adding those values, but we expected that. So let's apply it, say is equal to, and then we'll look at the data frame, and this looks almost exactly what we're hoping for; we just need to get rid of these, so this n-Dash and this na-Dash, we need to get rid of those, and that is super easy to do. Um, we're just going to say, so now that we've done it, and we'll comment that out, we'll say DF, and let's copy this, ignore the messiness, I do apologize for that; it's very messy. Um, but if you're following along with me, you get what we're doing. So DF phone number, so only on the phone number, say do string.replace(parenthesis). Now we can specify this value, so we want to take this exact value and replace it with nothing, and let's just see if that does work. It does. Now we have these Nas, and so let's actually, I'll paste that right down here. We're going to do this is equal to, and then we're just going to take this entire string, put it right here, and put this value as our what we're looking for and then replacing, and then when we call that data frame, it should work properly, and it is perfectly cleaned. So we have every single value, all the exact same; they don't have different characters or different um, you know, formatting, and we got rid of all the ones that we don't have or don't need, um, all the ones that were just random values. So this column is now completely cleaned up. Again, definitely one of the more difficult ones, um, one that I've done a thousand times. I've had to work with a lot of phone numbers and stuff like that. This one does get very tricky, especially if you have like a plus one, which is like an area code, um, that can get tricky as well, but this is on a kind of a high level; this is how you can do that, and it's pretty neat how you can actually, you know, clean up and standardize those phone numbers.

So let's go right down here. Uh, let's run it. The next thing that we're going to look at is this address. Now let's just pretend that the people who are on the call center want all these separated into three different columns; they can read it easier, see what the ZIP code is, where they live, uh, you know, whatever they want it for. Let's just say we want to do that, and this is, you know, again for this use case it may not make sense, but you have to do this; I do this all the time. Um, you need to split those columns. Now luckily all of these things are separated by a comma, so we can specify that we're going to split on this column, and then we'll be able to create three separate columns based off of this one column, which is exactly what we want. Then we can name it as well, and we can do that very easily by using this split. So we're going to say DF, and we want to specify, oh jeez, not again. So we want to specify that we're looking at the address, then we're going to say do string.split, and we'll do an open parenthesis. Now the very first value that we need to specify is what we're splitting on, so we want to split on the comma, so we want to specify that, and then we need to specify how many values from left to right it should look for. Now we'll just start with one, and then we'll go from there. Let's just see what this looks like. So it doesn't really look like it did anything. Let's do two. Well, let's go back to one, and then let's say expand equals true. When we expand it, it's actually going to uh separate it, I believe. Okay, so we're expanding. Now we're only doing this with one comma, so we're only looking at the very first comma and splitting it, but in some of these, well, just in one, there is an additional comma, so we should do it up to two. Let's do this. Okay, so now we have three columns. If we just save it like this, it's going to give us these 0, 1, 2, these basically these indexed values for these columns, and we don't want that; we want to specify what these actually are, and we can do that by saying DF, and let me just do is equal to, we'll do bracket, and then within there we're going to specify our list, so we have three of them that we have, so I'm going to do um the first one; this is the street address, so we'll say street address. The next one is um, it's sh is not a state, uh, but these all are states, so I'm just going to say State, and then for the very last one that looks like a zip code, so we'll say zip, and we'll do re code. In fact, I also want to do Street_address. Um, so what this is now going to do is these three columns are going to be applied to these three names, and they'll basically be appended; it doesn't replace the address; we're not saying DF address equals the DF address; we're not replacing it; we're now creating different columns. So let's run it, and then let's also call it, so they're right over here on this right-hand side; I couldn't see them at first, but it did exactly what we needed it to do. So now if we wanted to, at the very end, if we want to, we're not going to, we could just delete this address and keep the street address, the state, and the zip code. Another really common thing that you can do, this happens often again with like first name, last name, where you have Alex freeberg, but it's Alex, freeberg or Alex space freeberg, and you can separate those out into different columns.

Now the next one that we want to look at is this paying customer, and the paying customer and do not contact are very similar, um, in the fact that it's yes, no, NY, yes, no, NY, um, and so let's go right on down here, and we're going to say DF. and we want to just replace these values as all yeses or all NOS, but just with the same formatting, um, just to keep it consistent. So let's make anything that's an N into a no, anything that's a a y into a yes; I like it spelled out, so let's change anything that's uh a yes into a y, anything that's uh an no into an N; that's usually how I do it; just saves on data because it's less strings, although it's you can be often very minimal. Um, but let's specify the M customer; we'll say DF[paying customer], then we'll do string.replace. So now we're just going to look for those specific values, so if it's a y, oops, a capital Y, then we'll say yes. Now let's run it, and now we have no more y; we now just have yeses, although now these are yes yeses. Okay, we don't want to do that. Let's do if we're looking cuz it's taking, it's literally looking up here and saying, okay, there's here's a y, um, let's change the let's change that Y into a y. So now it's doing ye yes, uh, we don't want that, so let's look for the yes and change it into a y. Now when we run this, that looks a lot better. Um, so we'll do DF paying customers equal to, and then we'll copy this; we'll do the exact same thing, no and N. Then let's call it, and now that entire column looks really good except for that value right there, but I'm going to leave that because I'm just going to apply it to the entire thing all at once to get rid of those at the end instead of just going column by column, and then it's literally going to be the exact same thing, so I'm not even going to scroll down. Whoops, I'm just going to put it right up here because this is the exact same thing; I'm going to save us all some time, and when we run this, this looks exactly like what we're looking for. Again, some not a number values, but we can get rid of that in just a second by doing a place over the entire data frame, and that is basically the end of cleaning up individual columns.

Now let's go right down here; we're going to say DF.string.replace, and then we'll first do these values, oops, so we'll do oops, let me do that there. We go and replace that with nothing. Let's just see what it looks like. Oops, data frame object has no value string. Well, that's cuz we were looking at columns before. Yeah, I think I just need to get rid of this string; we're not looking it; we're just really doing it across the entire data frame. Now let's try that. Okay, that worked appropriately, and we'll just say data frame is equal to, and then we'll copy this, and we'll do the NN as well, and we'll [Music] do, and now when we do this, it is not going to replace these because these aren't actually a value because we're looking for that string; we actually need to use, and I, I completely forgot this; I'm not going to lie to you. Um, let's get rid of this uh to get rid of those values because it's literally not a number; there it is technically empty. Um, I forgot we can do um, or we could not even specify it; we'll do DF.fillna, so we're going to fill these values; if there's nothing in them, we're going to fill it, and we're going to say blank, and when we run that, every value that doesn't have something in it is going to show up blank, even over here where we only had a few; all of them throughout the data frame, if it doesn't have a value, it is now blank. So let's apply that, and we'll run this, and now all of our cleaning, we actually cleaning up the individual columns is completely done. We've removed columns, we've split columns, we've formatted and cleaned up phone numbers, we've also taken values off of first name or or this last name column, and then we formatted in just kind of standardized paying customer and do not contact. Now they also asked us to only give them a list of phone numbers that they can call, so if we take a look, some of these do not contacts are y, which which means we cannot contact them, and then there are some that don't even have phone numbers, so we don't want to give the people the call center numbers that or or people who don't have numbers, so we want to remove those. Now there's a few different ways that we can do this, but let's start with, and we'll just go by do this do not contact; it seems like the most obvious one. Now if it's blank, we want to give them a call; we only want to not call them if they've specifically said we cannot call them, so if it's y, we're not going to call them. So what we need to do, it's not anything like this; we probably need to loop through this column and then look at each row that has a value of this and drop that entire row, uh, and we probably will need to do that based off this index instead of doing it based off just this column, uh, that may not make sense, but let's actually, let's actually start writing it. So we'll do for X in, and we need to look at our index, so we're just going to do, let's do nf.index, and we'll do a colon, enter, and then we want to look at these indexes; how do we look at these indexes? We use loc; that's going to be df.loc, and then we need to look at the value, which is this X right here, so each time it looks at the index, it's looking at the value, but we want to look at the value of this column, do not contact. I don't know if I copied this before; let me copy it. We only want to look at the value in this one column; if we didn't, it would look at um a different value, so we don't want that. So we're looking at just that value; if it's equal to Y, so if this value is equal to Y, then we want to drop it, so we actually need to say if, so if this value X in this column is equal to Y, then we want to do df.drop, and then we'll say X, and we I think we have to say inplace equals true here, otherwise it won't take effect, um, otherwise you have to say like DF is equal to DF that, yeah, I don't I don't want to start messing with that; let's just do inplace equals true, um, and let's see if that works. I, I can't remember if this is going to work or not. Invalid syntax. Okay, neon. And now let's try to run this. Okay, okay, yeah, if we look at our index, we can already tell that there are ones missing; the one, the one is missing, the three is missing, um, let's see, and the 18 is missing, so we already got rid of those values, and you can you can see that there's no y's in here anymore, uh, which is really good. We can, if we want to, and we probably should, we should probably populate that, um, really quickly. Um, let me just go up here really quick; I'll copy this. We probably should populate that, and I didn't plan on doing this, so um, if it's blank, oops, it's blank, given an N, and we want to attribute it to do not contact, do not contact, whoops, let's see if that works, and we probably need to do do string. Let's just see if it works. So if it's blank, dud. Okay, I don't know why it's giving us a triple N; maybe there's maybe I need to strip this or something. Uh, okay, never mind. Let's not do that, but now we basically need to do the exact same thing for this phone number, um, because if it's blank, we don't want them calling it, um, so we can copy this entire thing, go right down here, and but now we're looking at phone number, so now we're looking just at the values within phone number, and we only want to look at if it's blank, so if it literally has no value, we want to get rid of it. Let's run this and see if it works. Again, it should. Good. And now our list is getting much smaller, so you can see in our index a lot of um those rows were removed, and okay, good. Actually, this worked itself out because these all have ends. Um, so right now we're sitting really good; everything looks really um standardized, cleaned; everything looks great. I might drop this address; if you want to, you can drop this address, but besides that, this is all looking really good. This Paint customer doesn't uh the yes and nos aren't really anything. Um, now we could, and we probably should, before we hand this off to the client or the customer call list, we probably should reset this index because they might be confused as why there's numbers missing or, you know, they might use this index, um, to show how many people they've called or I don't know, something like that. So let's go right down here; we're going to say DF., and then we'll do reset_index, and let's just see what this looks like. Um, it does work, but as you can tell, it didn't uh get rid of that index completely; it actually took the index and saved that original one. We do not need to save that. Whoops, let's put it right in here. Now we're just going to do drop equals true, and when we do that, it just completely resets it, drops the original index, and gives us a new index, and that is what we want. Let's do DF equals, and this is our final product. Now one thing that I you definitely could have done here, um, and I made this a little probably more complicated than it needed to be, um, that was just how my brain was working at the time when I'm, you know, typing this out. We could have done DF.drop, and a um, which is literally going to look at these null values, um, before we couldn't do that with this one because these aren't we're not looking at na; we're looking at y's, so we couldn't do that, but because we're looking at null values, we could have also done dropna, um, and done subset is equal to, and then done it just on this phone number, and then done like this, and done inplace equals true. So we could have also done this, then said DF equals, I can't, I mean, I can run it; it's just not going to do anything; I can run it on the different column, but that'll mess everything up, but this is another way you can do it, and I'll just save it in case you want to. Um, I'll say another way to drop null values. There you go, and that'll just be a note for us in the future. Um, but this is our final product; it looks a lot different than when we first started. I mean, we had mistakes here, completely different formatting in the phone number, different address, everything that we just talked about, um, and this looks just a lot, lot better, and you can tell why it's really important to do this process because again, we're working on a very small data set; I, I purposely, you know, created this data set with these mistakes because, you know, when you're looking at data that has tens of thousands, 100 thousands, a million rows, these are all things that are going to be applied to much larger scale, and you won't be able to as easily see them. Um, you'll have to do some exploratory data analysis to find these mistakes, and then you're going to need to clean the data or doing it at the same time when you're exploring the data, uh, so you'll clean it up as you go, but these are a lot of the ways that I clean data, a lot of the things that you can do to make your data just a lot more standardized, a lot more um visually better, and then it really helps later on with visualizations and your, you know, actual data analysis. So I hope that that was helpful. I know that this is a long video; I'm sure it was, uh, but I hope that you got something out of this, and you learned some of the techniques on how to actually clean data in pandas. If you like this video, be sure to like and subscribe; check out all my other videos on pandas as well as Python, and I will see you in the next [Music] video.

Hello everybody, today we're going to be looking at exploratory data analysis using pandas. Exploratory data analysis, or EDA for short, is basically just the first look at your data. During this process, we'll look at identifying patterns within the data, understanding the relationships between the features, and looking at outliers that may exist within your data set. During this process, you are looking for patterns and all these things, but you're also looking for um mistakes and missing values that you need to clean up during your cleaning process in the future. Now there are hundreds of ways to perform EDA on your data set, but we can't possibly look at every single thing, so I'm just going to show you what I think are some of the most popular and the best things that you can do when you're first looking at a data set. The first thing that we're going to do are import our libraries, so we'll do import pandas as pd. We're also going to import

Seaborn and matplotlib. Now, during this exploratory data analysis process, I often like to visualize things as I go because sometimes you just can't fully comprehend it unless you just visualize it, and it gives you a larger, broader glimpse of everything. So we're going to import, and let's do seaborn, oops, as sns, and then we'll import matplotlib.pyplot as plt. Let's run this; that should work. Okay, perfect.

Now we need to bring in our data set. So we've worked with that world population data set; that is the exact one that we're going to use now. So we'll say dataframe = pd.read_csv(), and we'll paste in our CSV, and this is what it should look like, although your path may be different. Be sure to make sure that you have the correct file path. Then we'll read it in.

Now, this data set should look extremely familiar if you've done some of my previous pandas tutorials, but I did make some alterations to this one. Took out a little bit of data, put in a little bit of data here and there, um, to change things up. Because if it was just exactly how I pulled it—which I got this data set from Kaggle—if it was exactly how we pulled it like we've looked at in the previous videos, it's too simple; you know, we wouldn't actually be able to do some of the things that I would like to show you. So be sure to actually download this exact data set for this video because it is a little bit different. But what we're going to do now is just try to get some high-level information from this.

Now, if yours looks just a little bit different, like your values are in scientific notation, uh, I have applied this so many times; I think it's, um, you know, still applied to this. You can do something, and we'll write it right down here: we're going to do pd.set_option(), and we'll do an open parenthesis, and we'll say display.float_format, and so we're going to change that float format by just saying lambda x:, and then we're going to change basically how many, um, decimal points we're looking at. So let's just do here, so we do a quote sign, ".2f". So we're formatting it, whoops, ".2f". So we're going to format it, and we'll do %x. This is going to format it appropriately. I'm—I can run it, um, and actually it will change it because this is at 0.1, I believe, last time I did it. So let's run this, and then let's run this again; it'll change it to 0.2. So that's two. I like it at 0.1. We don't really need it any—well, let's keep it at point two. Why not? We're going to keep it at point two. That's how you change that, and I like looking at it like this a lot better than scientific notation. So just something to point out.

Um, let's go down here and let's just pull up dataframe. So we have this data. One of the first things that I like to do when I get a data set is to just look at the info. So we're going to do .info(), and this gives us just some really high-level information. This is how many columns we have, here are the column names, here how many, uh, values we have, and if you notice, this is where it kind of gets—so we have 234 in each of these. So in each of these columns, we have 234 until we get to this 2022 population. Once we get there, we start losing some values, and then at the world population percentage, we have all of our values, all 234 of them. The count tells us that it's non-null, so it does have values in it, and then we also have the data types, and these come in handy later, um, and these are really great to know, and we'll be able to kind of use those in a few different ways later on in this tutorial.

Really quickly, I wanted to give a huge shout-out to the sponsor of this entire Pandas series, and that is Udemy. Udemy has some of the best courses at the best prices, and it is no exception when it comes to Pandas courses. If you want to master Pandas, this is the course that I would recommend; it's going to teach you just about everything you need to know about Pandas. So huge shout-out to Udemy for sponsoring this Pandas series, and let's get back to the video.

The next thing that I really like to do, and this one is df.describe(); this allows you to get really a high-level overview of all of your columns very quickly. You can get the count, the mean, the standard deviation, the minimum value, and the maximum value, as well as your 25th, 50th, and 75th percentiles of your values. So just at a super quick glance, there is a row somewhere in here and there—this country, their population is 510 for 2022, and in fact, if you go back to 1970, it was higher; was at 752. That's just interesting. Then if we look at the, um, max population, one has 1.42 billion; I believe that's China. And then over here in 1970, we have 822 million; again, I still believe that's China. But this gives you just a really nice high level of all of these values or all these different calculations that you can run on it, and we can run all these individually on even specific columns, but, you know, is just a nice high-level overview.

One thing that we just talked about was the null values that we're seeing in here. Um, I'd like to see how many values we're actually missing because that is a problem. Um, we don't want to have too many missing values; that could really obscure or change the data set entirely, and so we don't want that. So we'll say df.isnull(), and then we'll do a parenthesis, and we'll say .sum(), and when we do this, whoops, .sum(), there we go. When we do this, it's going to give us all the columns and how many values we're actually missing. Now, we have 234 rows of data, so we have 41, 477, 55,424, um, so we have—we definitely have data missing. What we choose to do with it in the data cleaning process—maybe we want to populate it with a median value, maybe we just want to delete those countries entirely if the data is missing. Um, you know, I don't think you're going to do that, but—but these are things that you need to think about when you're actually finding these missing values. This is what the EDA process is all about; we want to find different, um, either outliers, missing values, things that are wrong with the data, or we can find insights into it while we're doing this as well. So this is definitely something that I would consider, um, when I'm actually going through that data cleaning process. Really, really important information to know.

Now let's go right down here, go to our next cell, say df.nunique(), and this is going to show us how many unique values—and it's actually nunique—this is going to show us how many unique values are actually in each of these, uh, columns. And this one makes the most sense, um, for continents because I think there's only seven continents, right? Um, but we have six right here. And for all of these—each of these ranks, countries, capitals—should all be unique; that makes perfect sense, as well as these—you know, these populations are such specific numbers and such large numbers; I would be shocked if any of these were similar. And then for these world population percentages, it's much lower, and again, that makes a lot of sense because when we're looking at—and we'll pull it up right here—when we're looking at these world population percentages, um, a lot of them are really low: 0.00, 0.01, like this one, um, 0.2; there are a lot of really low values for those small countries, and so those are all, um, you know, one unique value.

Now let's say we just have this data right here, and we want to take a look at some of the largest countries, and we can easily do that. We could even—we could say max and take a look at the largest country, but I want to be a little bit more strategic; I want to be able to look at some of the top range of countries, and we can do that based off this 2022 population. So we'll say df.sort_values(); this is how we sort and, um, not filter, but, um, order our data. So we'll do sort_values(), and then we'll do by=, and then we'll specify that we want this 2022 population, and then we're going to say comma, and we'll say—actually, let's just run this as is, um, but we'll do .head() because we just want to look at the top values. So now we're just looking at the very top values. So what we're looking at is actually these 2022 population; that's what we're filtering on or sorting on, basically, and we're looking at the very bottom values because it's sorting ascending, so from lowest to highest. So this Vatican City in Europe is, um, you know, 510; that's the value that we were looking at earlier. Now we can do comma ascending=False because it was by default True; we can do False, whoops, we can do False, and then it'll give us the very largest ones. So if we just take a look at the top five largest by population, we're looking at China, India, United States, Indonesia, and Pakistan. And we can even specify that we want the top—top 10 in this head; we can bring in the top 10, and we also have Nigeria, Brazil, Bangladesh, Russia, and Mexico. And you can do this for literally any of these columns, whether you want to look at continent, capital, country, um, you can sort on these and look at them, and you can even look at, you know, things like growth rate, world percentage. This one seems really interesting; let's just look at this one really quickly before we move on to the next thing. Um, if we look at this world percentage, just China alone—I believe, yep—just China alone is 17.88% of the world. So 17.88%, again, just getting in here, looking around; that's all we're really doing.

Now I want to look at something, and I have always liked doing this, which is looking at correlations, um, so correlation between usually only numeric values. We can do that by saying df.corr(), and a parenthesis, and we'll run this, and what this is is it is comparing every column to every other column and looking at how closely correlated they are. So this 2022 population, if we look across the board, it's very highly—I mean, this is a one-to-one; this is highly correlated to each other, and that almost for all of these populations, they're very, very closely tied to each other, which makes perfect sense because for most countries, they're going to be steadily increasing, and so they're probably almost exactly correlated. But we can look at these populations, and if you look at the area, it's only somewhat correlated, and that's because in some countries, you know, they have a very high population but a small area, or vice versa, a small area in a very high population. So there isn't a one-to-one correlation there, but it's hard to really just glance at this, um, and understand everything that's there. We could just visualize it, and it would be a lot easier. So let's go ahead and do that. Let's go down here; we're just going to visualize this using a heatmap, basically. So we're going to say sns.heatmap(), and an open parenthesis, and the data that we're going to be looking at is df.corr(), and then we also want to say annot=True. I'll kind of show you what that looks like in just a little bit, um, but let's do plt.show(), and this will be our first look. And I need to say show, not shot. Um, we can get a little glimpse of what it looks like, but this looks, um, absolutely terrible. Let's change the figure size really quickly. So I want to make this much larger than it already is. We'll do plt.rcParams['figure.figsize'] = [20, 7]. Okay, that looks a lot better, and, um, you know, this is just a quick way because it gives you basically a color-coded system: highly correlated is this tan, all the way down to basically no correlation or negative correlation even, which is black. So when we're looking at these 2022 populations, and these are populations right here, on here, on this axis, we can see that all of these are extremely highly correlated, very, very quickly, whereas the rank really has nothing to do—it's—it's negatively correlated; doesn't really have anything to do with it. Then for the population and the world population percentage, it again is quite correlated, except for the area, density, and growth rate. So I find that really interesting that, you know, the density, the growth rate, and the area aren't really all that associated or correlated with the population numbers. That is—I kind of would assume that on some level they went hand-in-hand. The area does—would, you know, again, make sense—sense, you know, larger area, larger population, that kind of thing. But even density, um, I guess—I guess density and growth rate—growth rate I can see because that's a percentile thing that could be definitely not correlated, but I thought the density would be more correlated than it is. All that to say is this is one way that you can kind of look at your data, see how correlated it is to one another; that can definitely, um, help, you know, what to analyze and look at later when you're actually doing your data analysis.

Let's go right down here. Um, something that I do almost all the time when I'm doing any type of exploratory data analysis like this, I'm going to group together columns, start looking at the data a little bit closer. Um, so let's go ahead and group on the continent. So let's look at it right here; let's group on this continent because sometimes when you're doing this EDA, you already know kind of what the end goal of this data set is, you know, kind of what you're looking for, what you're going to visualize at the end; that really comes in handy when doing this. But sometimes you don't; sometimes just going blind. And so far, we've really just been going in blind; we're just throwing things at the wind, kind of seeing some overviews, um, looking at correlation; that's all we've done. Now I kind of want to get more specific; I want to have like a use case, something I'm kind of looking for, not doing full data analysis, not diving into the depths, but something we can kind of aim for. So the use case or the question for us is: are there certain continents that have grown faster than others, and in which ways? So we want to focus on these continents; we know that that's the most important column for this use case, this very fake use case, um, so we can group on this continent, and we can look at these populations right here because we can't really see growth; you can see a growth rate, but the density per, uh, kilometer, we don't have multiple values for that; it's just a static one single value; same for growth rate, same for world population percentage, but we have this over a long span, many, many years, um, you know, 50 years of data here. So this—we can see which countries have really done well or which continents have really done well. So without, you know, talking about it even more, let's do df.groupby(), and then we'll say continent, oops, let me just copy this; I'm—I'm not good at spelling. We're going to say df.groupby(), and then we'll do .mean(), and we can just do it just like this, and now we have Africa, Asia, Europe, North America, Oceana, and South America. Okay, so if I'm being completely honest, I knew most of these—all right, I'm no geography expert, but I—I knew most of these. I don't know what this Oceana is; this—that I don't—I genuinely don't know what that is, um, so let's just search for that value and see. We'll come back up here in just a second, but I want to—I want to kind of understand, um, what this is. So we're going to df, um, and we'll say continent, let me sound that out for you guys, um, then we'll do .str.contains(), oops, contains, good night, and then I want to look for Oceana, uh, and let's—let's run this. Oh, I need to do it like this. Now let's run this. So now we're looking at our dataframe; we're seeing when the values have this continent as Oceana, um, okay, so these look like islands, I'm guessing. So we have Fiji, Guam, um, New Zealand, Papa New Guinea; yeah, these look like all—I'm—I'm guessing based off the continent Oceana, Oceania, o-Ocea-Oceania, guys, this is tough for me. Okay, I'm doing my best; I—you know, this is part of the EDA process; I don't know what that means; I don't know what ocean-ocean-ocean-Oceania; geez, I'm just going to call it Oceana; that's so wrong, but I'm just—so easy for me to say, you know, I—I now am seeing this, and it looks like islands, um, which would make sense because for their average, they have the highest average rank, um, and I'm guessing that's because they're just mostly small continent. So let's—let's order this really quickly. We're going to do .sort_values(), do an open parenthesis, and I want to sort on the population; we're just doing the average population; we'll do by=, um, equal, so on the average population, and we'll do ascending=False. So we're looking at this average or the mean population. Asia has the highest population on average, then we have South America, Africa, Europe, North America, and then Oceana at the very bottom, which makes perfect sense—again, small islands, um, world population percentage. So each of the countries, each of those countries in Asia, makes up about 1% on average. Really interesting, um, to know and just kind of look at this, and the density in Asia is far higher than double—almost double—every single other continent, um, really, really interesting actually, now that I'm looking at this. But, you know, that's something that I would actually look into, and I would be like, what is this Oceana or Ocenia, what does that mean, and, you know, let me look into that, let me explore that more because I want to know—this data set, I'm trying to really understand this data set well. But what I want to do now is I want to visualize this, um, because I just feel like looking at it—I don't—it's hard to visualize. And again, the use case that we're saying is—is which continent has grown the fastest; like, it could be percentage-wise, it could be, um, you know, as just a whole, on average. Let's take a look. So we're going to take this, and let's copy it like this; let's bring this right down here. So let's look at this. So if I try to visualize this, and let's do that, let's do df2 = because I'm—I already know it's not going to look good just based off how the data is sitting, um, we can do df2, oops, what am I doing? I don't need to do that, but I will. Okay, df2, and we'll do df2.plot(), and we'll run it just like this, um, as you can see, Asia, South America, Africa, Europe, North America, Oceana; we can kind of understand what's happening, but these are the actual, um, values that are being visualized, not the continents, which is what I wanted, um, in order to switch it, and it's actually pretty easy, and this is something that, um, you know, is good to know; we can actually transpose it to where these—these continents become the columns, and the columns become the index, and all we have to do is say df2.transpose(), and we'll do this parenthesis right here, and let's just look at it, and then we'll save it. So now all these columns are right here, and all of the indexes are the columns. So we'll say df3 =, and I'm just doing—doing that so I don't, you know, write over the df or my earlier dataframes. So now we have this dataframe three. So now let's do dataframe3.plot(), and it should look quite a bit different, uh, whoops, I didn't run this; let's run this and run this, and as you can see, this does not look right at all, and the reason is is because we're not only looking at, uh, the

Correct columns. We have this density in here; we population percentage rank. We don't need any of those; the only ones that we want to keep are these ones right here: this population. Now we can do that, and we can just go right up here. This is where we created that data frame two that we transposed. We can go right up here, and we can specify within this; we actually only want specific values.

Now we can go through and hand-write all of these, and by all means, go for it, but I am going to go down here. I'm going to say `DF.columns`, and I'm going to run this. This it's going to give us this list of all of our columns, and I'm just going to—you can just copy this, and you can put it right in here. Think I need a list with—I think it needs to be like this. If I'm—let me try running this. Okay, so this worked properly. You can do it just like this, or a little shortcut if you want to do it like that. If you want to do a shortcut like, um, I—I would hope you would—you would just do `DF.doc.columns`, just like how we looked at down here, except—except since this is our an index, we can search through it, so we can just say 0, 1, 2. Okay, so we can do five up to 13 because I think it's seven, and we'll just—let's see if this works. Uh, it may not; I may actually need to go like this. Let's see. There we go. So you can just use, you know, the indexing to save you some visual space; gives you the exact same output. So now we have this. This is our `df2`. Now let's go down and transpose it.

So now we just have these populations, and we have our continents right here, and then now we're going to plot it. And this looks good, although it's backward. Um, okay, it's backward. So what I actually want to do is not this. Uh, that is a quick way to do it, although not the best way to do it. Um, so I'm actually going to copy all of these, and although I said it would save us time, it did not at all. So I'm going to put a bracket right here; I'm going to paste this in here, and I'm literally going to change these up. I might speed this up, or I might just have you sit through this because, you know, this is an interesting part of the process, and I want, you know, you to get the full experience. You know what? Now that I'm talking about it, that is what we're going to do. You guys can hang out with me; this is a good time. We have 2010, 2015, 2020, and 2022. Now let's run it. What did I do? Oh, too many brackets. There we go. So now it's ordered appropriately. We have 1970 all the way up to 2022. This is how we want it. Let's transpose it appropriately. Let's run it, and now we basically have the inverted uh image of this. Now, just add a glance, and we haven't done anything to this except for literally what we are looking at. At a glance, we can see that from 1970, China, you know, Asia and China are already in the lead by quite a bit, and it continues to drastically go up, especially in the 2000s, like right here; it explodes, like just straight up, then kind of starts going up and just leveling off. Every other continent, especially Oceana, is just really low; it—it never has done a bunch. Let's see. Look at green; green has gone up um from, you know, point—let's say 0.1 up to about 0.2, so they've almost doubled um in the last 50 years. And again, you can just get an overview, a high-level overview of each of these, you know, continents over the span of this time. So this is kind of one way that we can, you know, look at that use case. We're not going to harp on that too long; I just want to give you an example, like, you know, when you're looking at this, sometimes you'll have something in mind of what you're looking for, and you go exploring and just kind of find what's out there and find what you see.

Um, the next thing I want to look at is a box plot. Now, personally, I love box plots; you know, they're really good for finding outliers, and there's a lot of outliers. I already know this because the average, the 25th, 50th percentile are very low, and then there's some really just big outliers. But for your data set, it may not be that way, and those outliers may be something that you really need to look into. And box plots have been something that I've used a lot where I found those outliers that way and started to dig into the data to find those outliers and, you know, came across some stuff that I'm like, "Oh, I have to clean this up; I have to go back to the source," really. Really, really powerful and useful to be able to find these. So all you have to do is `DF.boxplot`, and let's take a look at it. And this already looks good as is. Maybe I'll make it a little bit wider. Um, let's do `figsize`; oops, sorry, `figsize` is equal to—let's try 20 by 10. Um, okay, that didn't help at all. I apologize; thought it would, but let's keep going. What this is showing us is that these little boxes down here, which are actually usually much larger because you have a more equal distribution of—of um, numbers or values in the small value—this is where our—our averages lie. This number right here is the upper range, and then all—all these values, all these open circles, those actually stand for outliers. So we're looking at the 2022 population; there's a lot of outliers. Now, for our data set, knowing our data set is really important; outliers are to be expected, especially when most countries or continents are small. So we're looking at, you know, all of these little dots are outlier countries um or outlier values, which each value corresponds to a country. So if this was a different data set, I would be, you know, searching on these and trying to find these so that I can see what's wrong with them, if anything, or if they are real um numbers. Like, if this was revenue, everyone's revenue is way down here, and then there's one company that's making like 10 trillion dollars; that'd be an outlier up here, and it would definitely be something that you want to look into. For our data set, knowing that, you know, we're looking at population, this is more than acceptable, you know, oddly enough, but that's what box plots are really good for—showing you some of those core tiles, the upper and the lower um as well as denoting these points that fall outside of those normal ranges for you to look into. So really, really useful. So now let's go down here, hold up our data frame again, and we've kind of just zoomed into the whole EDA process. There was one last thing that I wanted to show you. This is the very last thing that we're going to look at. We're ending on really a low point, if I'm being honest, because the last kind of stuff was more—much more exciting, but there is something: `DF.dtypes`. Oops, let's do `DF.dtypes`, and we'll run this. Now, just like `info`, it gave us these values, but we're actually able to search on these values now. So these—object, float, and integer—we can search on those, which is really great because we can do `include=` and we can use something like `number`, and none of these are numbers, right, or none of them explicitly say number, but when we run it, I'm getting an error: `series object not`—oh, that's because I'm doing—`dtypes` is for a series; we need to do `selector.dtypes`. Now let's run this. Now it's only returning um the columns in this data frame where the data types are included in this number. So you won't see any, you know, country or any of those text or the strings. If we want to do that, we go in here and say `object` and run that, and this is another really quick way where we can just filter those columns to look for specific—whether it's numeric—we could even do `float` in here, and so now it's not including that rank, which was an integer. So we can specify the type of data type, and it'll filter all of the columns based off of that, which, you know, when you're doing stuff like this, you—it is good to know what kind of data types you're working with and look at just those types of data types because there might be some type of analysis you want to perform on just that—whether it's numeric or just the string or integer columns within your data set. So again, ending on a low note, I apologize. Um, you know, everything else that we looked at, all those other things that we looked at are all things that I typically do in some way or another when I'm looking at a data set. Exploratory data analysis is really just the first look; you're looking at it; you're going to be cleaning it up, doing the data cleaning process, and then you're going to be doing your actual data analysis—actually finding those trends and patterns and then visualizing it um in some way to find some kind of meaning or insight or value from that data. And again, there's a thousand different ways you can go about this; it—it does typically um, you know, depend on the data set, but these are a lot of the ways that you'll clean a lot of different data sets, and so, you know, that's why I went into the things that we looked at in this video. So I hope that you guys liked it; I hope that you enjoyed something in this tutorial. If you like this video, be sure to like and subscribe, as well as check out all my other videos on pandas and Python, and I will see you in the next video. [Music]

What's going on, everybody? Welcome back to another video. Today we are back with another data analyst portfolio project where we will be scraping data from Amazon using [Music] Python. Now you may be asking, "Do I need to know web scraping to become a data analyst?" And the answer is no; you absolutely, ely don't need to know it, but it is a very cool skill to learn, and in fact, I have used it in my job in the past, and so it is useful, but you really don't need to know it. Something that it is used for is kind of creating your own data sets, um, and we're going to be looking at one where you can create your own data set today, but there are a lot of other uses for web scraping, and I'm sure I'll talk a little bit more about that while we're actually walking through the project. One last thing I want to say before we get started is that this is most likely an intermediate project, so if you are just now learning the basics of Python, this might be a little bit challenging for you, but I still recommend going through it because I will do my best to walk through everything, every single step of the way, and—and kind of explain all of the concepts, and so you can still learn something even if you aren't super good at Python right now. With that being said, let's jump over to my screen and get started on the project.

All right, so we are going to get started, and if you didn't watch the last project, I had people download Anaconda. Uh, we use Jupyter notebooks, um, and I'll show you how to get to that in just a second, but I'll—I'll leave this link in the description if you haven't done that already and you are just doing this project, um, but you'll go—you'll download Anaconda. You know, download—super easy, um, and you're going to open up Jupyter notebooks. I'll launch it right now; I already have it open, uh, but I'll open up another one just for, you know, the purposes of demonstration. What we are going to do today and what we—what people voted on—I mean, there's like—there was like 8,000 people that voted um in the poll that I made of what data you wanted me to scrape; there was like Amazon, cryptocurrency, weather, um, something else; I don't remember. Overwhelmingly—I mean, like 70% of people, maybe even 80%, I—you know, don't—don't fact-check me on that—voted for Amazon, um, and so I'm going to do it now. There are many things that you can scrape um off of Amazon—just a ton of stuff—um, and I'm going to show you how to do it; I'm going to show you how to make it useful, how to make a data set, um, and it's going to be really interesting, but there are lots of other ways to do this, and so I think—and I have already kind of created it; I'm going to show you how to do it off of this page. Um, when you're actually in an item, and you can scrape, you know, basically anything in here, um, and I'll show you how to do that. Another thing that is a little bit more advanced, and that's why this first video is starting off, I think, on the more easy side—it's not easy, but it's easier—the next thing, the next video that I'm going to make is how to actually do um basically do multiple items, right? So this item, this item, this item, this item, and then traverse through the different pages. So there's 20 pages; um, you want all of that data; how do you get all of that? That'll be the next project. Um, I don't know when I plan on doing that; I have it like 90% of the way done, um, but I had this one completed, and so I wanted to get that out to you guys now, but that will probably be the next project. I think that is much more difficult, um, and so if you can understand this one and you get it and—and—and you understand it, then the next project you should be able to understand, too—is just a little bit more complicated. So with that being said, um, we are going to actually get into the project. I'm going to delete one of these. Um, all we're going to do is go to New, do Python 3; it'll open up a new one. We'll call this um Amazon web scra—um, project; that's what we'll call it. Did I spell it right? Perfect. Um, the first thing that we need to do uh or that we should do is upload um or—or—or import our libraries. So I'm going to say um `import`. Oops, what am I doing? Off to a terrible start. There we go. `import` libraries. Now I'm not going to write out all the libraries; um, I have some things that I'm going to be copying and pasting throughout this. I won't—there's only a few things that I'm copying and pasting. You can take a quick glance. Um, some of the things that I just don't want to waste time on um because this could be a long video. I don't know; I don't want to waste time on stuff like this, um, and so, you know, I'm just going to copy and paste it. You guys are going to—I'm—there will be a link below if you haven't clicked already that will go to the GitHub page where you can literally have all of this code already written. I do recommend writing it all yourself because you will learn it much better; I promise, because then you'll make mistakes and you'll figure it out and all that—all that good stuff, but you will have that code available. So just go copy and paste it; um, that's what I would do. But what we are—we are going to be using today is uh something called Beautiful Soup, requests, um, then we're going to be using `time` and `datetime`, and a potential one if you want to get—and I'm going to show you this at the end; this is not really part of the project; it goes above and beyond—but this library right here is for sending emails to yourself, um, and I'll show you how uh you can use it if you want to. I already have the whole code written out; um, you can just steal it and try it out yourself and see if you can get it to work, but this one is not um as important. I'll put it down here, so—um, let's move on. Now one thing I want to say before we get to into it is that—well, give me a second—is that right here in front of me is a different laptop. Now it took me a solid, I would say, you know, 10 hours or so to write all of this; it took over the course of like two weeks in my free time; I'd pick it up; it took me a solid, you know, two weeks on and off, an hour here, an hour there, to finish this project, um, and I made a ton of mistakes and messed a bunch of things up, and I finally got it to work, um, you know, after a bunch of revisions; that's typically how things go when I do projects, and so uh I'm about to give you a streamlined version of this because I have all the code right down here, and so I'm going to be glancing at this a lot, um, just so I don't make this video 20 hours of trying to remember all the code off the top of my head. I have it written out already; I already did the project; it works; it's beautiful; it's a good project, so—um, I don't want to waste your time, and I just want you to know that, you know, you—you—nobody should be able to do this off the top of their head in an hour; most people won't. It takes time; you make mistakes, um, but uh let's get started on the project. Now, in this—in this, what we're going to have to do is we are going to have to tell Beautiful Soup and requests where we are actually getting this data from—what website, um, what is our computer—you know, some information from our computer. I'm going to—again, there's going to be a little copying and pasting in here because you don't ever—you will never ever ever need to know this, um, but right here we're going to basically connect to the website, so I'm just going to say "Connect to website," and we are going to say `URL =`, and let's go get our URL. So we have this right here, so literally just go up here, do, you know, uh, Ctrl+A, copy that. Oops, that's the actual project; get rid of that; uh, paste it in here, and that is our URL; we will use that in just a second. Uh, what am I doing? Me—just get some room here, and then we—what we're going to need is something called headers. Now, again, you will never ever ever need to know this, so I'm just—just going to say headers. Um, what I'm going to do is I'm going to copy this; I'm going to show you how to get this really quick, um, but is something called headers. So uh let me show you how to use—how to get this and why you don't need to know any of this. So what this headers is is this something called a user agent. You need to do this for your computer, um, and you can do that by going to this link right here. So I'm going to put this link in the description so that you can go and get that, and there's something right here called the user agent. So all you have to do is copy this, just like this, do copy. I'm going to go back here, and I'll show you that it's—I'm going to copy it in; it'll be the exact same. So there you go; it's the exact same—all of this extra stuff except encoding, except—this HTML stuff, connection close—all the—you don't need to know any of it; I promise; you'll never come in handy ever in life. Actually, there will be one person who that becomes in handy for, and then they'll message me. Um, but we are now connecting um using our computer, using this URL, and then what we want to write is we want to write `page =`, and this is where we start using uh these libraries. So we're going to use `requests.get`, and we are going to pull in that URL, and we're just going to say `headers =` our headers right here. So uh we have this, and this is where we're going to actually start getting the data, bringing in the data, um, and it's not going to look like that at first, but I'll try to print some stuff out as we go along the way so that you can kind of see what it looks like and how we're going to kind of—

Make it more useful because it comes in very dirty, uh, when we first get it. And some of the things I'm going to show you will just help clean that up. Um, and before we actually go any any further, I don't want my head to be here for the entire time. I'm going to get rid of myself so you can just see the page. Uh, I just—it's less distracting. Uh, I hate when I feel like people are always watching me, so I want people to just focus on the code. Uh, so I will see you in a little bit. Let's get back into it.

All right, so what we are going to do is we are actually going to start using the beautiful soup Library. All right, so we are going to say soup one is equal to, and this where we actually start bringing beautiful soup, and you guess it, you're going to say beautiful soup. And then in parentheses we're going to do page.content. Um, and again these aren't really things that you need to remember or need to memorize. We're just pulling in the content from the page; that's really all we're doing right now. And it comes in as HTML, so we're going to do html.parser. Uh, and let's see if I can print out—uh, actually let me just do soup one. I don't like I don't like doing upper caps on stuff. Let's see if anything prints out real quick. So we are literally pulling in all of the HTML. Um, and let me go show you really quick because we're going to get to this in a second anyways. Um, if you come here, this is this is a static page basically written in HTML. Um, if you have never seen HTML before, um, you know, actually a lot of this is, you know, just stuff that most people will never use. Uh, it's just good to know; some of the stuff is good to know. So as you see I'm scrolling on this right side; by the way, I did right-click and inspect, or control shift I, whichever one works better for you, but as I'm scrolling over this, you should see it kind of highlighting different areas. Um, it's hard to kind of get what you want. Let's say we want this title. Um, what I can do is I can click select element, go right here, um, and then we can select like a TI, the the the header or the title of the the page. Now I just wanted to show you though of what we're pulling in, so we're pulling in this doc type HTML; all of this is coming in. So that's what this is right here, this doc type HTML, and we're pulling every single thing in; that is what we're doing right now. Uh, so let's get or let's go down a little bit. Let's do soup two. We're just going to do a very, uh, you know, an upgrade to soup one basically. We'll do beautiful soup again, and then we're going to do uh soup one, so we're pulling in that content again, so that's soup one, and we're going to do .prettify. If you don't know what that is, it is common in a lot of different languages and a lot of different stuff. Um, it just makes things look better; that's really all it is. Uh, I don't know why I'm using double quotes; I don't know why I—you can do single ones if you want. Um, and now let's do beautiful soup two, and it should just be a—it should be better formatted. Um, and let's see if that's true, and it is. So before, if you did—if you can tell, it was didn't have basically any formatting; it has a little bit of formatting now. Um, it'll help in a second. Um, and you'll see that. But now what we want to do is go back, and we want to actually get the data that we want. Now you can get any data you want. I'm going to show you simple things, really really easy, um, in my in in in my opinion, it gets more difficult the more complicated stuff you start pulling, um, and and you'll understand that as we go into it. So what I'm going to do is I'm going to select this and I'm going to select this—the title, I want that. And so if you do span ID, it's equal to product uh title, so we need to remember that. Um, class, we don't need to know class, I believe. Uh, we're going to be using that ID, this—ID equals product title, so that's what we're going to be using. Um, class will come in in the next video when we start looking at these, uh, but not in this one. So let's remember ID equals product title. So let's go back over here. So we have this soup two; it's basically all of that HTML in it, right down here; that that is what we were pulling in. So we need to kind of specify what we actually want. So let's say title; that's what we're going to be getting. Um, and we're going to do soup 2, so using—taking all that content—we're do find, and we're going to do open parentheses; we're going to say we want to find that ID where it's equal to product title, and then we're going to do .getText, and then we're going to do open parentheses. So now let's—let's print the title and see what we get. All right, so that is exactly what we're looking for. It's funny, got data Mis—T-shirt; that that is what we're trying to pull in, so that's perfect; that's exactly what we want. We don't—let me let me just do this, save me some time later on. We don't only want the title; we are also going to be pulling in the price. So if you can guess, uh, we'll be doing some—a data set on the actual pricing. Um, and so let's go back here. We're going to again use this right here, and we're going to go to this price, and it says—again we're going to look at this ID; the ID equals price block or our price. Did I say that right? Excuse me. There we go. And the exact same thing.getText(parentheses. Uh, and there's a getText; there's a getAll or getAllText. Um, so you know that getText is a specific thing that we are using; you we might use a different one later on, um, but that that is what we have. So now let's let's print the title and print—why do I have all this too much—too much space. So let's print the title and print the price. Let's see what we get. Okay, so we have our title and we have our price. I mean, you know, I don't know what all this white space is over here, um, but it looks like there's a lot of white space over here. We'll have to get rid of that uh in a little bit as we clean it up a little bit. You can, if you want, do things like—you can get, and this is up to you, I'm not going to do this right now, but I'm just going to show you how to do it. You can get this where you're pulling in the ratings, um, which is, you know, if you want to look at like how the ratings over time or or what ratings are for specific products; that could be really useful. You can pull basically anything; you can go down the product details and look at Dimensions, uh, anything you want on this page; it is static, so you can go in here and pull anything; it's you just have to pull it from the HTML, know where you're looking, pull it in. Um, and now when we go back here, excuse me, I'm going to show you now kind of how to use this right, because we have this, but how are we going to use it? Um, that's kind of the important part, I think. First thing we need to do is clean this up a little bit because it just is, you know, if we try use this, it wouldn't be super useful because it'd be—it's just a little bit dirty; it's not super clean. Um, so what we want to do is let's start with the price; why not? Uh, we're going to say price.strip, um, and that's just going to take uh basically the the junk off of either side. And so let's run that real quick. So this is what we have, but what we can also do is I don't want that dollar sign; I just want the numeric value. Um, later on we are going to be putting this, and we're going to be—creating a process to put this into an Excel file. Again, we're trying to create a data set; I don't want you to have to copy and paste stuff; it's all going to be automated basically to input this data into an Excel file for you or a CSV file for you. So, um, you know, think about making it useful in a CSV or in an Excel later on. So what we can do is do a bracket, and we're going to do one, and then everything after that. So basically it's just going to take everything from the first position onward. Uh, so let's run that, and there we go. So let's just say price is equal to price.strip, um, and pull—just do everything after that first—that first not value; what am I saying? What's the word for that? I can't remember the word; the first space; that's not the right word, but all right. Let's do the title. Um, this is basically going to be the exact same thing, super easy. So we're just going to do title.strip and open parentheses. Um, and we can, you know, if you want to do this exact same thing. So now we have it; it's a little bit cleaner. So this is what it originally looked like, and now this is what it looks like. So, you know, nothing super crazy, but you know, something interesting to know. Now we are about to—in the very next part, what we are going to do, and let me just add a few of these because it makes me feel better—what we are about to do is we're going to create our CSV to insert this data into the CSV. And then later on what I'm going to do is show you kind of how to—automate this process to pull this data—to create a data set, right? Just pulling this one time and putting it into a CSV really doesn't do anything; you can just copy and paste that and save yourself a lot of time. Um, what I'm going to show you is is—basically doing it over over time and just having it automated in the background; that is what I'm going to show you. Um, I guess a spoiler, but what we need to do is we need to create—create the CSV, insert it into the CSV, and then create a process to append more data into that CSV. I'm doing a lot of talking; let's do some writing. So what we need to do is we're going to use—I should have done this at the top; maybe I'll go back and add that later on. We're going to do import csv. Now in a CSV, what you want is you want headers, and then you want the data, right? So for our headers, and we're going to call it header, we're going to do—we're going to do a bracket, and let's make the first one a title because that's going to be—we can call it title; you can call it product, whatever you want. I'm just going to call it because I've been using title; I'm going to call it title. Um, and then we'll also have price. Now we need our data, so I'm going to say data is equal to—now this is important—right now how our data is, and I can do this right here, we're going to do type—title or no, let's do type price. So these are strings, and that's important to know. Um, again, I don't want to get too much into, you know, dictionaries and arrays and lists and and strings and all these things, but this is a string, and you can't put that—right now it's not super usable. What we're going to do is make this a list, um, and so I'm doing an open bracket, and I'm going to say our data is title, comma, price. Oops, price. Now oops. If I do type—oops—of data, I'll just run that; it's a list now. Um, and this is important because you can run into a lot of issues with this stuff; it's really important to remember what's what type—how do I say this—how your data is; is it a list? Is it an array? Is it a dictionary? Um, you know, what is it? These things are important; they do play a big impact, especially with this type of stuff. So just wanted to show you that really quick. But what we are now going to do is create a CSV. You're going to create an Excel—I call an excile—CSV, you know, whatever you want to call it. So what we are going to do is we are going to say with and we're going to say open, and now we're going to name our file; you can name this whatever you want. I'm going to call it uh—Amazon web scraper data set; that's real long—.csv, and then we're going to do _w, and that means WR—oh, whoops, that's not right, just like I was wondering why that was uh in black—so we're going to do w, which means write, um, and then we're going to do newline, and if you don't know what newline is, uh, all that does is when we insert the data, it doesn't have a a space in between each CSV, and then we are going to do encoding is equal to—oops—is equal to utf8, and that is it, and we'll just say as uh let's do f. So some of that stuff you don't need to know; some of it's useful. This w definitely need to know; this newline is is good to know, and—I'll take it—I might take it out just to show you what it actually does because it's annoying if you don't have it, I promise. Um, but you know that that newline's important; this encoding, you know, good to know; I think that's by default is is it's like that. Anyways, what we're going to do now is we're going to—it's something within the CSV—within the CSV—the—Library, so we're going to do something called csv.writer, and oops, csv.writer, and we're going to do open parentheses, and that is that, and we'll just call that writer, and then we'll do—this is where we need to actually create the header, so—uh, we're going to do writer is dot—sorry, writer.writerow, uh, and this is just for the initial—the initial import or or or—not import, the initial insertion of the data into the CSV; this is what's important. The next one that we're going to write is for when we're actually appending the data, which is going to be a little bit different, but anyways, we're going to do writerow, open parentheses, and this is where that header is going to go, so we're going to—these headers are going to be the title and the price. And then for our last one, we're going to actually write the data, which is this data right here, and we're going to say writer.writerow, and we're going to do data. So this one we are creating the CSV, and then we are inserting the header and inserting the data, so super easy. Um, yeah, I think that's fairly straightforward right now. Let's do this, and let's see what happens. So I just ran it. Um, let's go over here; in here somewhere, Amazon webscraper data set; let's open that up, and there we go. Oh geez, this isn't good. Can't verify my—my subscription. Uh, why does it say $6.99? Uh, I'm going to go back and look, but I think I know the issue, um, but this is exactly what we want. Now, of course, we want more data, and maybe a little bit more useful data, um, and I'll show you how to get that in just a second. But we just created that out of thin air; that was not—I didn't have that saved before. So we have this data set, and the issue was is that I ran this multiple times, so now it's $699. If I do it again, it's 99, uh, and if I did it again, it gets rid of everything. So I'm just going to run this again, run this again. Uh, now everything's back to normal. Okay, so now if we run this, it's going to overwrite this Amazon webscraper datas set.csv, and it will put the data in properly. So there we go. Oh geez, guys, this is embarrassing. I'm embarrassed. No, I don't want this. Okay, perfect. Um, guys, I—if you can't tell, I'm in need of some—I'm in need—I'm in need of some help here, but I'm just kidding; I'm I'm doing fine; I just—I don't know why that—why I don't have my—uh subscription activated; it's not going to matter for this video, I guess, but that's really random. Um, so we got what we need; that's perfect. Now what we want to do after this—I—I guess actually what is important is some more useful data. Something that I like to do a lot when I do this type of this type of stuff is I like to have some type of date stamp, um, or some type of time stamp to know when I collected this data; it usually comes in handy later on. I I have never regretted putting it in there. I'll show you really quick how you can do it. Uh, you're going to do import datetime. Geez, I hate having to format stuff like that. And what you can do is you can do date—let me get datetime, and you do date.today(parentheses, and that is going to give us this right here. Uh, and so we're just going to do—today; that's what we'll call it, is equal to this, and we'll say print today, and there we go. So that is today's date is the 21st of August in 2021. So today is now—is now this. So actually I'm going to get rid of that; I'm going to put it back up here; I'm going to—going to put it right there. I'm going to run it again. Let's add this right here; we'll do—we'll do—we'll call it date, and then we'll add today, and we'll just run this again. And what we can do, just to check the data without having to open up the data every single time, which is super annoying, is we're going to use pandas again. I should have imported this at top; I'm just kind of—I'm not doing this off the top of my head, but—I didn't have it 100% planned, so import pandas, and we're just going to say pd.read_csv, and then we'll read it in. Um, what you can do or what I often do is I go to properties, and I go right here, and we'll say boom boom backslash this right here; this I am doing off the top of my head; I don't do this often; I think I have this memorized by now. Uh, I I I hope. And then we'll do print—oh no, we don't have to do print; we'll just do this—what do I do? R—let's actually call this—data frame, and we'll do print data frame. Let's see what happens. Perfect. Okay, so what we have now is the new—our new header, our new data that we added in there, so we have our title, we have our price, and we have our date. Now, again, you can customize this whatever you want to add; go back here. Um, you know, find what you want; you know, do you want it to make sure it has a men's option or different colors or you want to pull in this information? Whatever you want; it it really does not matter. Um, just matters that you know you get what you need for whatever purpose, whatever you're making this for. This is more of an introductory video to how to scrape data from Amazon. Um, the next video will probably be a little bit more difficult and in-depth, but this is kind of—let's get you guys started. So—we now have this, and this is beautiful. Now something that you want to do when you're scraping data and you're getting—I guess data over time, and that's kind of what we're doing; we—it's going to be almost like—a price tracker over—

Time is you want to then append data to this, so we can't only create it, and that's what this does. Because if I run this 100 times, it'll only give me this first row. We need to now append data to this, so um, let's let's pull this down here. U again, I'm I'm not I haven't added a bunch of notes. I'm going to say now we are appending data to the CSV. I haven't added a ton of notes; I'll try to go back maybe afterwards and add some notes for people who like to read notes. Um, so what we are now going to do is we're going to change this W to an A+. Now this is going to be how we append the data, um, and we no longer need the header, so we don't aren't going to do the header anymore, and there we go. So now instead of, excuse me, so now instead of creating that header again, creating that first row of data again, we are ignoring the data and we're now going to the next nearest free row and appending data, which means to add on data to that. Um, and so if I run this, which I'm not going to right now—oh, I mean, why not? I can I can run it—um, and then we can read this in. So now there there's our data. I'll run it a few more times. I ran it like three or four more times. I I run that in, and there we go. Now it's all the exact same data, super um, boring, but very very uh, you know, good to have.

Now we don't want to have to come in here and run this every day, let's say we're going to do this daily. Um, we don't want to have to come and write run this every single day, right? We want a way where it does it while we sleep, it does it in the background of our laptop, um, and is easy to do, right? I don't want to come in here every single morning with a set an alarm on my phone every single morning, come in here. I want to automate this. So uh, how are we going to do that? Give me one second. Uh, if you didn't know, I have three kids, and one of them is waking up. I be right back. All right, I think he is asleep, um, at least let's hope he's asleep. So now what we're going to do is we're going to put this all into uh, this check uncore price. Now you may never have used—oh gez, what are these things called? Oh my gosh, super use all the time. You'll know what I what it is—uh, not a function. I don't even remember what it's called. Maybe this is a function. Um, I can't think; I'm having like a writer block or whatever that is. We're going to put it all in here, and then we're going to be able to use this price check later, um, because we want to be able to automate this. So let's go back all the way up here. We are going to use this, so let's copy all of that in, and oh jeez, I hate this. All right, everything just like that. Um, so this pulls in our data, pulls in uh, or or yeah, pulls in all of our data down to the title and the price. We want to make it look right, so we're going to put it right here. So now we have it formatted properly. Um, we want to add our date time, it just like that. I don't know if there's a better—I'm sure there's a better way to do this—um, then we need this right here, and just like that, like that. So now we have our header and our data, and then we want to pull this in right here. Boom boom boom. Okay, so everything that we just wrote out we are now putting into this check price. Now you can call it whatever you want, doesn't matter, but let's run that, see if we get any errors. We don't. So this is now good to go basically.

Um, what we are going to use this for, um, and what this is going to do is we are going to put this on a timer. Um, you know, have you ever wanted to like check something once a day, once every 10 seconds, once a minute, whatever you want, and you don't want to have to actually pull up your phone and look at it? This is how we are going to do that. So we had something called uh, let's see, time, this this Library time right here, that's what we're going to use right now. So we're going to say while oops, while true, and go like this, do a colon. We're going to say check unor price, that's what we just wrote out, and we're going to do time.sleep. Now this is completely up to you how much time you want to put in here. For the purposes of demonstration, I'm going to put 5 seconds, which means every 5 seconds it is going to run through this entire process. And so let's run this really quick, and I'm going to run it for, let's say, 30 seconds, and then I'm going to pull this in right here. So we just looked at it earlier; we had four—well, five rows of data, right? What we are going to do is in just a second I'm going to stop this, you know, maybe after 30 seconds or so, we're going to see how much data is in there, uh, and let's stop it right now. It's we can going far enough, um, and now let's run it. So now we have 5, 6, 7, 8. So I guess I ran it for 20 seconds. We can that was for demonstration purposes. I've never do any some anything every five seconds, um, unless it was like Black Friday on Amazon. We can put this as long or as short as you want. You can run it every second if you want. Um, that doesn't make sense to me, but you can. What we can do is do a little bit of math, uh, and I don't know this off top of my head, so I'm going to uh, do the math with you live. Pretty exciting stuff. Got the calculator out. So there are 60 seconds in a minute, and this goes by seconds by the way, and you could do you know you can do some um, some string up here of calculating this, but I'm just going to put in the number because it's easier, uh, maybe not easier. I'm just going to do it. There's 60 seconds um in a minute, there are 60 minutes in an hour, so that's 1 hour, uh, and we can do 24 hours in a day, so that that's 86,400, I believe. Did I read that right? Oops, did I read that right? Yes. So this now, if I ran this, and I'm going to—this is going to check the price every single day, and this is the entire point of this—of of this project, not the entire point, but this is a big part of this project is we want to create our own data set. Now something that I personally really love is a data set that has, you know, that I can do some type of Time series with. Now this is not exciting; it's probably not super exciting for this right, but you get the idea that if this price were to change, we would then see that reflected in the data at some point. You can do this on any item you could ever imagine on Amazon; it's the exact same process, and some items change often. This t-shirt will most likely never change, um, and so, you know, again, this is for demonstration purposes. The code itself will be nice to put in a project, although the data set that you get from this probably won't be the best I would imagine. But notice that the is running. Um, I can then minimize this, and this can run on my computer basically as long as my computer uh, is is working.

One thing I will say before I go on to some more stuff, one thing that I will say is that I personally, when I did this for a when I um created this, I did something similar and I put this in Visual Studio Code, um, and I didn't put it in Jupyter notebooks. That's a personal preference. I would look into that if that is something that you want. Um, I think Visual Studio Code is a little bit easier for automating these types of tasks, um, but for illustrative purposes and for demonstration purposes, you cannot beat Jupyter notebooks. That's why I did it. So with all that being said, that is basically the end of the project now. Um, I'm not going to stop this and read it again, but you get the point. Um, we now have um a data set that—oh jeez, all this again—that now has um data. I'm getting out of here. Oh jeez, it's hounding me. Let me get out of here. Oh no, all this is embarrassing, guys. I'm embarrassed. We now have a CSV file with data in it. Now you run this in the background of your computer; you can do that. I have done it; I've ran it for weeks; I have ran it for months. Um, if you restart your computer, just come back in here and restart running this process. Um, it's the same for any automated process unless you start using some online um automation service, which will run it regardless of your computer; they do it, you know, either in the cloud or on some um server. So you know that this is a really good option again if if you restart your computer or something happens, you lose connection, just come in here, run this through the script again, um, except for the one where it deletes all your data; don't run that one again. Only run that one time, um, and then you will—in fact, what I would do is then um, I would just comment this out, right? I'd come in here, and I would just comment this out so that anytime I come back in here, I would never accidentally delete all my data. But that is what this project does.

Now something really interesting, something that I have done in the past that I thought was really cool, really useful, I actually did did it for um, I actually did it for some watches that I was watching, especially on Black Friday; it's when I used it. I was interested in a price drop or a specific price change, and what I did was is I said—and I don't know—so what I basically did was is I said if the price is lower than, let's say, let's say we wanted to drop below $14, it would then send an email, um, and I'm going to show you the script that I used; it still works, um, and if this is something that you are interested in, this could be a completely different project. I just think it's interesting, and I wanted to show it to you, although I wouldn't say this this is part of the um final project. Let me just come in here, and we are going to create this super simple—well, not super simple; we're sending a mail; we're connecting to a server; we we're using Gmail; we're logging into our account; that is my email; you will not get my password; we're creating the subject, the body, um, we we configure or or just kind of create this message, and then we send a mail. So then I have this Define uh, or this send mail; I am blanking on what this is called. I'm going to call it a function, but that's probably not right. So if that price drops below a certain point, it'll send me an email. Um, I have used this, and I used it and was able to buy a watch that was like, you know, let's say 140 bucks for like 90 bucks, um, on Black Friday sale. I was really really happy about that. So this can be used in that way as well, um, not something you have to write into your project, just something I'm going to include down here if you want to try it. I think it's super interesting, something really fun, um, really fun to mess around with. I enjoyed this. So with that being said, uh, this is this is the project. Um, I in the next one, and I promise you this one is probably going to get a lot more difficult. If you thought this one was easy, which I hope maybe I hope you do, then that means you're, you know, pretty good at Python. You know, in the next the next um web scraping project, and I hope to do many of these; I might do um even all the ones that I put in that poll, but I started with the one that was the most popular, um, you know, if you were able to get through this, I think that that is fantastic. I think this is a solid project to create um a data set and so use this how you will. You can copy my code exactly; I don't have a problem with that. Again, I don't think this is beginner; there are some a little bit more advanced things in—I don't even Advanced, just like intermediate level things—um, that you kind of learn as you get into it, and so um, I hope that this was instructional; I hope that I explained it, you know, well, um, and I hope that this is useful. Again, you know, when you actually use this, you'll have 22, 23, 24, 25, you know, you'll see a price change, a price change, a price change, a price change. Go use a product or go to something that you were interested in or that you know fluctuates often, um, and there are plenty of those on Amazon; I promise you there are some that literally change almost every other day, like down a dollar, up a dollar, um, and then Black Friday just goes crazy um with these price changes. So use this as you will. I hope that this was instructional; I hope that it's useful. I think I said that before; it's you know, I'm doing this because I think it's really interesting; it's really useful, um, this to me again was a good introduction, a really good introduction to web scraping, because in this next one it gets quite a bit more difficult. Um, I would say on a scale of like difficulty, this is like maybe a four, and it'll probably jump up to like a seven on this next one, um, just just much more um technical or coding heavy, so um, you know, look forward to that if that's something that you look forward to. With that being said, I'm going to go back over here for my send off. With that being said, I hope this was helpful; I hope that you learned something. Um, don't get mad mad at me if it was too easy; don't get mad if was me if it was too hard. Uh, I'm doing my best over here, so I appreciate your patience. Thank you so much for watching; I really appreciate it. If you like this video, be sure to like and subscribe below, and I will see you in the next [Music] video.

What's going on everybody? Welcome back to another video. Today we're going to be creating a script to automatically take data from a crypto [Music] API. Now this project stems from an earlier video that I did where I walked through what an API was and how you can use it, and in that video I showed you how to use coin market cap's API so you can start pulling in their crypto data. And in this video we're going to take it one step further and automate that process. Now we're going to do a little bit of transformation with the data; I'm going to show you some cool stuff of how you can use it, and maybe we'll do a little bit of visualization at the end, but that is not the main point of this video; it's mostly around the automation piece and a little bit of the data cleaning piece as well. Now fair warning, this is not a beginners level project; it's probably more like an intermediate project, and it's not even a complete project per se because we're not doing all the data cleaning; we're not doing all the visualizations. But if you follow along, we're going to cover a lot of different things, and you're really going to set yourself up to be able to do just about anything you want with this data or different APIs that you pull from. So with that being said, let's jump onto my screen scen and get started with the project.

All right, so this is where we stopped in our last video. So if you haven't watched it, now is the time to go back and do that. I'll have a link in the description; also, all the code that we're going to be looking at today and working through is going to be in a GitHub repo below, so you can go and get all the code and have it completely finished and just follow along, or you can code it from scratch along with me. I do recommend writing it from scratch if you can because I think you'll learn more, and you'll make mistakes, and you'll learn from that as we go through it, but it is up to you. So let's get started, and as you can see, uh, we have this script right here, and I'm starting basically from scratch. I have a completed one up here; I'm actually going to get rid of those, um, and what we're going to do is we're going to start from exactly where we started in our last one. I'm going to run the script; um, this is going to pull from our API, and we're going to look at the dictionary, set our option, and do our Json normalize. So this is where we literally left off from the from the last video. So we have all of this data, and what we want to do with it is we want to kind of automate that process, right? Because we don't want to have to come in here, run this, and you know, put into a CSV manually or something like that. We want to automate this data collection process so that we can just have the data ready for us to use, um, and it all be ready to go. So we're going to be using this script, um, but you know, we we might want to add a little bit more to it before we do that. Uh, the first thing that I want to do before um before anything is something that I like to do when I'm creating these automation scripts is I I like to add a timestamp, uh, and the reason for that is because I want to know when I ran or when each of those um loops you can say runs through an and does those automated runs, right? So if I do it every day, I want to know what time of day I ran it, making sure each run ran successfully. And so all I'm going to do is I'm going to add a new column at the end, just call it timestamp. So let's go right up here, and we're going to say PD Dot, and there's something called to date time, so we're going to do twocor date time, and then we're going to do now, and what this is literally going to do is take the the date the the timestamp of right now when it's running and it's going to show that. Now we need to of course add a new uh, a new column for that. So all we're going to do is we're going to say data frame, whoops, we say data frame, and let me see real quick, we just have the data we need to add; we need to create this data frame right here. So data frame equals and then this Json normalized, and we're going to say data frame, and then we're going to do a bracket, and we're going to say Tim stamp, and we'll do—well, are all these lowercase? We're going to keep with the the lower case; we're going to say timestamp, and we do that bracket, and we'll say equals. So what this going to do is going to first off it's going to create this dat or assign this DF as our data frame, and then we're going to add this timestamp and add this new column. And so let's run this really quickly, and let's go all the way to the right, and this is our Tim stamp, and this is the time that it is right now; this is the day that I'm running it; this is the time that I'm running it, and so this is working properly. Now if you look really quickly, there is a last updated in here, and this is very close to this timestamp, but it is not the same thing, um, but if you looked through this data and you really dug into it a little bit, there's this last update is coming from coin market cap's API, and this is when the actual um cryptocurrency was updated in their system, and so it is going to be really close, but it's not going to be exact, and so I don't like to rely on built-in ones that you know are coming from an API or something. I want to

Make one myself that's running on the system where I'm creating the automated process, just like just something I do. Um, so now we have this original data frame created, right? We have what we need, but what we want to do is to keep adding data to this. Um, we don't want it to just go to, um, you know, create these 5,000 rows; we wanted to create 5,000, 5,000, 5,000 over time, whether it's a day, an hour, a week, um, whatever you want to run it. So, um, what I'm actually going to do is I'm going to limit this a lot. I just want to look at the top, let's say 15. So we do that; we're going to run through all this again. So now I just have the top 15; it's going to be um easier to see and it won't take as much time to run our scripts again. You can keep as many as you'd like; if you want 100, 200, all 5,000, you do whatever you'd like. But what we are now going to do is we're going to create a function using this original script. So again, we have this data frame, and we are going to create an automated process that is going to automate a script to automate this that is going to append data to this data frame right here. So that's kind of, you know, the big thing that we're trying to accomplish in this project. Um, so let's go up here, and we're going to—we'll just take from here all the way to here—we're just going to copy this and going to paste it down here. Now what we need to do is we need to create a function, so we're going to say DF, and we're going to call this the API uncore Runner because this is going to run our API um whenever we need it to run.

Now, when you are formatting um something for a function, it needs to be formatted properly, and so what we need to do is need to go over here; we hit tab; we're going to do this all the way down. I'm just going to skip forward when it's all the way done. All right, so now we have this URL, and what we want to add—because this is again, this is going to run through kind of this this automated process—we're going to run this um this function there; what we want is to also add this right here. So we need to take this, and we're going to need to add this; we'll just put it down here. [Music] Okay, let's do that. So what we have so far is really close to what we want our function to be. Um, we have this function that we're going to be running through; it's going to call this function; it's going to call the the API; we're going to use our key; we are going to um, you know, test it, load it, format it, and format it right here. Then we're going to add this timestamp, and then we will have this. Now, right now it's just call—it's just going to print this data frame basically, but that's not what we want right now. What we want is to actually append this data. So when it gets to here, when it gets to this data that's going to be right um right here, what we want to do now, since we already have the original data frame set up up top, is we now want to say that this is going to be data Frame 2, and we're going to say it's going to append it to data Frame 2. And so the original data frame, we're going to say data frame 2.append, and we're going to say df2. All this does is this says this new data that's going to be coming in every time—let's say it's a loop, and it's just looping through, pulling the data, pulling the data, pulling the data—we're going to create this data frame, we're going to add add this timestamp like like we want, and then we are going to append that to this original data frame. So as of right now, this looks good. I will—we'll run it in a second. I'll create it. So I just created it. So now we need to actually create our script to automatically run this. So we're going to do something called import OS, and let me tell you, there's a thousand different ways to do this, and there are better ways to do this, but they are much more complex, much more complicated, and some cost money in order to do it. I'm going to show you different options on how to do this in future videos on how to autom your Python scripts, but this one to me is one I've used a lot um many many times for different projects, and it works. So I'm not going to show you the most complicated thing in the world; I'm going to show you something that I've just used a lot. And so we're going to say from time import time; from time import sleep; that one's important. And now we're going to create our loop. So what these um what the time and the sleep and the OS uh or your operating system—what what these are going to do is they're going to give us the ability to track the time, and we're going to be able to run through and call this function in certain intervals that we want. So let's create our for loop; we're going to say for I in—now you can create this specific part in different ways, but what I'm going to do is I'm going to say range of one uh let's say 333—and I say 333, and if you remember from the first video on the API, you only have 333 runs per day. And so if I ran ran this 333 times today, that would be our Max, and so that's why I'm using that 333 just for reference. So now we're going to do API Runner. So in this loop, we're going to call this function up here, and then I'm going to say I want to prove or or show have an output to show that this is running through successfully. So I'm just going to—and you can write anything here—we're just going to say API Runner completed uh completed successfully successfully. How do you spell that? Successfully. That doesn't look right. I'm just going to say completed. All right, forget that; I don't remember how to say uh spell successfully if that's if it's spelled it right. You guys spell it that way, but I can't remember. Now we're going to use this sleep right here. Now this counts it in seconds; you can change it to minutes, hours, whatever. We're going to have it run every minute, which is every 60 seconds. And so this is going to—I'm just going to say it's going to sleep for one minute, and then we're going to say exit. So all this is going to do—and this is again fairly simple; it's just simple for loop—and what it says is it's going to call this API, it's going to tell us that it ran successfully, and then it's going to wait for 60 seconds, and it's going to run again. That's it. So let's run this and see what happens, see if what we did works. So it ran the first time. Now I'm not gonna I'm not going to bore you because I'm doing this live. Exactly what we're about to get is what we're going to use. I didn't run it overnight or or for a week so that we have a bunch of data and what you were going to work with; I'm going to work with as well. So I'm going to wait a few minutes; I'm going to let this run. I want you to do the same thing. I'm going to let this run for maybe like five minutes or so, and we'll work with what we have, and we'll keep going with the project because again, we're not—the point of this project is not to create the final product or creating all the visualizations that um will most likely be in another video where we're taking all this data and doing all these things with it. The point of this video is to automate it, clean it up to where we have it to where we can really use it, and then I'm going to let you guys loose, and you guys can do whatever you want with it. And I think it's really setting you up for a lot of successful projects in the future that you can do all by yourself without me having to walk you through it. So as you can see, it's already ran through twice. I'm gonna pause for a second; I'm going to let that run through uh just a few more times, and then we will continue with the project.

All right, we are back, and of course, it's only ran what, five times? Um, it has not reached the limit of 333, so we are perfectly fine. What I'm going to do is I'm just going to stop this by clicking this uh square up here, and it's going to give us some error, and then we're going to check it, and we will see what we have. I don't know why it's taking so long, if I'm being honest. All right, so I interrupted it, and let's run this; let's see what we got. I hope we have more than 15 because if not, I'm going be very upset. Okay, so okay, well, uh, I made a mistake. Um, I was supposed to put data frame right here, and I had data frame too. So um, take change your script; do not do what I just did. We're supposed to be append—it's supposed to be data frame append, and we're supposed to be appending the original dat—this data frame two to the original data frame. So um, I messed up on that one. Let's rerun that; let's rerun that. Um, let's see: local variable DF reference before assignment. Okay, this is perfect because this happened to me before. Um, we're running into all sorts of good stuff. I like to keep this stuff in my videos. I laugh because I hate running into mistakes, but everybody says they they're happy that I do this, um, so I'm going to keep doing it. I'm not going to cut this out, I promise, um, but what we actually need to do is we need to go back up to this function because what happened was is we called this data frame, and now it's—it's because it's function—it's in what they would call a local variable. What we need to do is we now need to state that this is a global um it's just called a global; that's all it is. Um, and so what we're going to do is we're going to do tab; we say Global; say DF. And what this should do is this should declare it as a global variable, and it should let this run properly. Let's hope it does. All right, it's it's running. Um, again, I run into mistakes. Let me tell you something while we're here for just a second: this project, I ran into probably a hundred mistakes or a 100 errors, issues that I had to research for hours um and hours. I'm legitimately on Stack Overflow and just Googling and figuring figuring these things out. There were a lot of new things that I had never run into before um just on this project. And so um, everything that you're seeing is from after I went through all of those things or after I fixed all of those things and had to really work through them. It was it was very um it was frustrating at times; I just I couldn't figure it out. And so what you're looking at is kind of the polished version of that now that I have everything laid out because I I can't spend 10 hours on a project; nobody would watch it. So just know that if you are running into some of these mistakes or you run into mistakes later on when you're expanding this project, that's completely normal. So what we're going to do is we're going to let this run for a little bit, and then after maybe three or four minutes, we'll come back, and we'll keep going with the project.

All right, so let's run this and check and see if we have uh the data that we're looking for, uh, and it looks like we do. Let's go actually back up here really quick. Uh, we want to set this to display Max rows because I want to be able to see all the rows and not just um a few of them. So and that just instead of it gives us this scrolling instead of that dot dot dot that shows us just a few. So there's our original 15, then we have the next um the next loop, and then we have the next loop, and let me scroll over to the timestamps, and I'll show you what I mean. Um, so was rain on 526 1501. Let's go down: 526 at 150, 2905. I say 1501, 2905, and next one you can see was ran at 306 31. These are all the ones one minute after each other. My original one was from earlier: 32 33. Yeah, so you can see 32 31 3030 or um 3029, and this one was about 15 minutes ago when I first um ran the original data frame. Right. All right, guys, this is Alex from the future. I've actually completed this entire project in the video, and you're about to see all that after this, but I wanted to show you one more thing that you can do in this function up here that I didn't show you uh originally that I'm coming back to show you, and that's how to actually put it into a CSV. Now, all we've done in this one is we we've kept it all enclosed in a data frame, and that's it, and that may be great, but a lot of you guys are going to want to automate this and put it into a CSV, and I want to show you how to do that. All right, so what I'm going to show you really quickly is right here in this uh in this folder right here; I have all these different API 3es and 4s. These were tests that I did before, but what you can do is instead of just putting it into a data frame, you can actually append the data to a CSV and have that CSV sitting out there for you instead of just keeping it all in a data frame. And there's a lot of different uses for that; you may want to have that file separately from here just in case something times out or something breaks, which is a legitimate concern, or your computer shuts off or or something like like that—that is a legitimate concern. So what we're going to do is we're going to say um if not—and this is basically an if statement—we're going to say os.path.isfile. So what this is going to do is check if there's already a file under this name, and we're going to do r. or or R. If you have never done um if you've never done CSV stuff before, it's really important that you put that you you're going to get an error every time. So we're going to take this right here, and we're going to copy that, and we're going to put that right here, and then we're also going to do a slash, and then we're going to name it basically um let's name this API because I don't think I have that one in there; I think I deleted it. Yeah, so I don't have API, so I'm just going to keep it api.csv, and then I'm going to close that parentheses, and then we're going to add a colon right here, and we're going to say if that does not exist, we are going to write this to it and create it. So we're going to say data frame—that's this data frame right here—data frame dot we going say to_csv, and we're going to do that R, and then we're going to copy this, so let's just let's just replace it like that, and then we're going to say com header oops header is equal to column uncore names. So what this is going to do is if we run through this, and what we would have to do is um I'll talk about this in a little bit; we'll have to change this up a little bit, but what this is going to do is going to check to see if this file right here exists; if it does not, it is going to to create it and create the column headers based off the this data frame; that is what that does. Now what we want to do is say else, and this next part that we're going to write is saying if there's already the API file there, we want to append the data; we don't want to overwrite it or anything like that; we want to append the data. So we're going to say—we're basically going to copy this, maybe not the whole thing, but I already did it—so we're going to copy that, and we're going to say mode oops mode equals a, and a stands for append, and then we're going to say header oops keep messing up header; we're going to say false oops; we're going to say false, which means when it appends the data, it's not going to use those the column headers every time, which you don't want because every time you append it, if you added the headers every 15 rows, every 15 rows, you're going to have another headers that you're going to have to like go back out into that CSV and filter out and and get rid of them. So we're going to say header equals false. Now, just a second ago, I said you would need to mess with this just a little bit, and you would because every time um you'd be putting in this data frame, which it's already appending it to this data frame, so every time you'd be creating a lot of duplicates if you kept it exactly as is. What you were going to need to do is basically take it back to its to its um bones, um, so you need to kind of keep it like this. So what you need to do is just now run this, and it would work perfectly. Uh, let's test it really quick um to see if it works uh because I'm I'm promising you something; I want to make sure it actually works. Let's run it this time. Okay, so it just ran for the first time, so it should have created this file. Let's go see if that works properly. So now it just created that file, and now we're going to see if it actually appends the data. So let's wait just one time um and then I'm going to stop it; I'm going to see if it works again. I'm just verifying to make sure that what I'm telling you is actually working uh because if it doesn't, I would feel terrible; we don't want that. And while that's running, actually, I'm going to add this because now I want to show you how to call it. Super easy; we're just going to do pd.read_csv. Do that; we're going to call this just like that, and then we're going to say data frame, and we're just going to do 72, something random because I've already done this whole project; I don't want to mess anything up. So we're say data frame 72. So now let's stop this, um, and what we're going to do is once that stops, we're going to run this and see if it actually um worked and see make sure that this actually pulled the data in. All right, so we interrupted it; the file is ready to be read in. So let's read it in. There's our file. Um, let's see what did I mess up or did I mess anything up? Ah, I didn't mess anything up. This is the index for this file, and we already had this in here; we'd probably be able to get rid of it, but if you see, we have 0 1 2 3 4 5 6 7even 8 9 14, then we have 0 1 2, and if we look at the timestamp, it should be 1 minute apart. So it's 11945; it said 12045, so this worked exactly as planned. Um, again, you have two different options: you can just keep it how it was before, and I'll leave both of those options, you know, in the in the script so that you can kind of choose which one you want, but um, that's how you do that. So then right here, you're appending it to a CSV file, and then if you just keep this and you get rid of all this, you're just appending it to a data frame. Now, please continue continue with the rest of the video that I already have done, um, but again, I'm future Alex, so please continue with the rest of the video. Okay, so we have all this data; we have we have so many columns we can do now, you know, if you want to completely just go and do your own thing, you absolutely can do that. I'm going to mess around with a few things um kind of show you something that I did that I thought was really interesting um in order to visualize a data little.

Bit and transform it a little bit to make it more usable. Um, but we're not doing a full data cleaning; that's not what this project is. We're not doing a full data cleaning of this data; that would be a ma a very large undertaking because, honestly, this needs a lot of work.

One thing that I do want to clean up really quick, uh, is is this right here—this the math will be fine; it's just the way that it's shown on here is in St the scientific notation, and I don't like it. So what I'm going to do really quickly is just um get rid of that. So we're going to we're going to say pd.set_option, and we do underscore option, and this is going to be do parentheses; we going to say display. This is just this how this is formatted, so we're going to display uh .float_format, and we're going to say comma, and now we're going to use this Lambda: say x:, and we're going to say percent 0.5f and that right there, and we're going to say percent X. Now, if you don't know what lambdas is, lambdas are um I highly recommend looking those up. Um, again, this is not a beginner tutorial. Whoops, no such Keys display floor format; that makes sense. Uh, this is float. Yeah, guys, this is not a beginner's level. All right. Uh, you can't use the floor format; this is the float format. All right, so now let's take a look at this uh this DF uh this data frame that we have, so we're just going to hit DF, hit enter, and now our numbers are a little bit more easily readable. I prefer it this way; you do not have to do this. I'm doing this just because this is what I prefer.

So let's jump right into it. Um, something that when I saw this data, I was like, something that I really thought was interesting is this percent change of one hour, percent change 24 hours, 7 days, 30 days, 60 days, 90 days. If you're not in crypto or you don't do investing or anything like that, what this is going to show us is how—I mean, it's pretty obvious—how much the price of this coin has changed over the last hour, 24 hours, seven days. So as you can see, it's it's barely fluctuated over the past 24 hours, a little bit over the past um seven days, a lot over the last 30 days, 60 days, and 90 days: 20 minus 26%, minus 33%. We're in May; we just had a kind of a crash in crypto a couple weeks ago, so I mean this tracks right, but I want to visualize this, see this and kind of see um, you know, how this is going to look and how if I can gain any insight from that information and just having it all displayed for me. But in its current state, um, you know, we really cannot do that.

Now, another issue—not an issue, but another thing that we have to take into consideration—is we have Bitcoin right here; we have Bitcoin right here after different polls. Now, we just did it a minute after each other, but for your project, you may do it a a run each day, a run every hour or something like that, right? And if you did that, your data could be very different, and so you may just want to take this first one. But what I'm going to do for the sake of this project, I'm going to group them. So let's go down here, and we're going to say DF.groupby, and so if you've ever done something like SQL, uh, this is how you Group by in pandas. Basically, we're going to group by uh the name, so so on bitcoin, ethereum, other, so we're gonna we're going to do that on name, and uh I'm not going to I'm going to say sort is equal to false. Oops, I'm not going to sort it. Uh, you could say true there, but we're not going to, and I guess you'll see why later. We're going to do an Open Bracket, and now we need to choose what we're going to group by uh or what we're going to what columns we're going to have. So I'm going to do another Open Bracket, and I'm just going to copy and paste these. So I'm going to start right here at quote percent 1 hour, so I'm going to do boom and then go over one, and we're going to take 24 hours, paste that, comma; we have the 7-day, 30-day, and we're going to do like that, and I'm just going to do comma; I'm GNA do the same one, but I'm just going to manually change it to 30-day of that at the end. I don't know what that is. Uh, then we're going to do 60 days and comma, and we're going to do our last one, which is 90 days, and let's see what that gives us. Ah, doesn't give us anything. Okay, I know what's wrong here. Um, we forgot to add basically the what we're we have; we're grouping by something; we need to have like an average, a mean, a mode or something like that, right? So all we have to do is go to the end right here, and let's just do me; we're going to do an average, um, and so we're taking this number, so let's say this is for Bitcoin, so we're going to take this number in this one hour for every time it's Bitcoin; it's going to group them all together, um, and then it's going to average them. So in the past five minutes where it's been running, we're going to take the average or the mean of that. So let's run this again, and so now this is our output. Let's take a look. Oops, I meant down here. Let's run this now. Now what we have is all of these um cryptos; these are all 15 that we have, and this is the average um for this 1 hour, 24, 7 days, 30 days, 60 days, in 90 days. So now we have all of our cryptocurrencies over here; we have our percent changes up top, and then our averages um here as well. And so now what we're going to do is, you know, if you try to visualize this as is, doesn't really work because these percent changes are up here as columns, and we don't really want them as columns because that it just doesn't work for visual—for actually creating the visualizations; we really need these to be rows.

And so my initial thought when I was doing this was I, of course, I need to Pivot um, you know, if you've ever used a pivot like an Excel or or power Pi or something like that, that was my first thought, and I tried everything and I could get could not get it to work, and I almost gave up until I I ran across um something called stacking or a stack, and and so this was not something that I I I think I have used it before, but I I couldn't remember—to be being completely Frank, I couldn't remember how to do this—so I just did um once I saw what it was, I did Stack. Let's make that j4. You don't have to do this; uh, you can keep this all the original data frame. I'm just—I like for visual purposes; you can see like the progression that we're making, um, but I like to, you know, create its new data frame, and I can always go back and look at this data frame three um as we go, but you don't you don't have to do that; that's just what I'm doing. So now let's take a look at this. Now uh up here we had Bitcoin, and we had all these columns, and we had had these numbers as rows, but now we have all of these as rows as well. This how we have this is much much more usable, um, and if you've ever done something like pivot or the stacking before, you'll know that you you kind of have to do it if you really want to visualize this well, but um you because we just stacked it, it kind of changed it. So if we look at um let's look at the type of Let's do type of data frame three. This is before um before we stacked it; this was in a data frame, but now let's go and look at data frame four. So this is a series; this is no longer a data frame, so we have to remember that that's that's really important because we can no longer treat it as a data frame; it's now a series. So we want to get it back to a data frame; we don't want it to be like that because you can't really use it in this series. So what we're going to do, and let me just create a few of these so you can be up here better, so now what we're going to do is we're going to say data frame 4. and something called to_dataframe, so we're going to make this into a frame, and now we're going to specify the name, and it doesn't mean um the name like right here; we actually mean the name of these values right here. This is part of the stacking process in these columns or these two columns. So let's go right here, and we're going to call it let's just say values, and let's make this data frame five, and let's see the output. Whoops, for data frame five, and now so there's that values, and now this already looks a lot better, right? So it's in this it's in this more um this is already a data so this is a data frame. So let's look at type data frame five. So now it's in a data frame, but the issue is is that this name is kind of acting like a an index, which we don't want because we want to be able to use this, so it doesn't really have an index at the moment, so we need to give it an index. But typically when you give an index, you'll do something like um we'll say data frame.five; we'll do set_index, and then you'll do something like um name uh so let's just do six is equal to; we'll see we'll see what happens here; it's going to give us an error. Oops, what I meant is we're going to do data frame five bracket uh name; that's a column, right? We're going to do that, and it's basically going to say that that's not going to work, and what we need to do is what or at least what I want to do, and what we're going to do in this video is I'm going to create numbers; I really would just want it to be numbered 1, 2, 3, 4, 5; that's what I want, um, but we don't have that right now. I can't just will it into existence. So now what we're going to do is kind of create uh an index basically out of thin air. So we're going to do pd.Index, and we're going to say uh, you know, we basically want how many um rows are in here; that's what we want our our um index to be; we want it to count how many are in here. Now you can make this dynamic, and I it probably wouldn't be that hard, but I'm GNA take this super lazy route, um, and I'm just G to say let's do DF.5 or oops df5.count, and there's 90 values in here, so what I'm going to do is I'm going to do a range of 90, uh, and this is not uh I would definitely make this dynamic, but I'm again I'm just being being a little bit lazy. We call this index is equal to, and I'm going to put this Index right here. So now this is a number, so now it's going to literally Index this for us. Now I've ran into this issue many times, um, so what I need to actually do is to reset this index and then do it properly the first time. Uh, so let's do re let's get rid of this; let's reset this index, um, and it actually fixed itself. Um, so what was happening was is we were indexing something that was already indexed, and we were causing issues in a nutshell. So we reset the index, and now this is what it looks like, and this is exactly what we want; this is really how we wanted it formatted in order to for our visualizations. We have multiple rows for the Bitcoin; each of these columns is now a row with the value attached to it—exactly what we wanted. So um really quick quick, I for whatever reason it it makes that uh level one; I don't know why, but we're just going to rename that column really quickly. So we're going to do data frame 6.rename, and then we're going to do an open parentheses, say columns equal to; we're going to do one of these bad boys, oops, one of these bad boys, this this type of bracket, and we're going to say level_one and do a colon and then oops and then a colon and then we want to change it to, and I'm just going to call this the percent_change. So let's call this data frame [Music] seven. Again, you don't have to do that; I'm just doing it. So now this looks much much better.

Now let's try to visualize this one um because we haven't done any visualizations yet; we've just been messing with the data a little bit. I I, you know, I kind of want to see how we can use this; this is something that I personally am interested in, so um I kind of wanted to see visualize how these changed over these these time periods, um, but we need to um import some stuff in order to be able to visualize this. So we're going to import seaborn as sns, and if we need to, um we're going to import matplotlib as well. I don't know if we'll use it right now or at all, but um we're going to we're going to add it in here either way. So now those are added, and so what we're going to do is come right here; we're going to do sns.plot, and we're going to oops we're going to say the x axis is equal to, and we want to do this as the percent change, percent_change, and then we have the y axis. Now we want the y axis to be these values right here, say comma Y is equal to, and we're going to say values, oops, and then we're going to say comma, and we'll say we want to basically create a Legend um I guess you could call it; we're going to say hue is equal to name. Um, I'll show you what it looks like without it, and then you know you can see that we need that. We're going to say the data is equal to this data frame seven, dataframe seven, and then we are going to say the kind is equal to. Now let's run this and see what we get, and super quickly with just you know limited um inputs, here's what we have. Now this looks really good. We can narrow this down if we wanted to to a few less because there's a lot here, and there's a lot of colors, but again, and that's just because we have a lot of different stuff, but there's a few that are doing really well. I think this is Tron, um, and then we have a few that are not doing so well, but it's really hard to see; if you look down here, it's really hard to see this, um, and that's just because of the the column names. And so I actually want to change these column names or these values so that when we visualize it right down here, it it doesn't look like that. I kind of want this to be, you know, at least one good visualization you can take out of here. This is definitely not perfect or complete by any means, but you know you can take take that away from here. Um, so let's um I did Alt Enter, which adds another row; I could have just pushed plus I; that was kind of the lazy way. Um, what I'm going to do is I'm going to change these um these values in here. So how I'm going to do that is I'm going to do data frame seven, and we only want to look at this one column, so we'll do that right there, and we want to say .replace, and we're going to an open parenthesis and then a bracket. Now what we need to do is—I'm just to show you um one of them—I'm going to say this one hour, do that, oops, and then what I need to do is a comma, another bracket, and this is what it's going to change to. I'm just going to say one hour, oops, one hour, um, and we'll do this one really quick, and then I'm going to—I don't want you to have to watch me type all this out, but I'm going to go through and basically do all of this uh for those. Let's let's see this really quick, and so now as you can see that um the originally it said quote.USD percent change 1 hour is now only one hour. Now this didn't actually do anything; we need to apply it to this right here, so I'm going to say data frame 7 is equal to, and then we'll run data frame 7even again. So now that has actually changed that value. Now I'm going to go through, and I'm going to update that for every single one.

All right, so I basically just put the other ones um in here that we wanted to change with commas afterneath, so I have 24 hours, comma, with the seven days, 30 days, 60 days, 90 days, and then this bracket over here, which tells uh it what to change it to: 24, 7 days, 30 days, 6 days, 90 days. So let's run this; I haven't even tried it yet, uh, and it looks like it obviously worked properly. So now let's go back down here, and let's run this again, and look at that; it looks so much cleaner, so much nicer, um, and as you—I mean, all of them with that one-hour change has very little change, and then you can look back, so we can see back within 90 days; it's gone; a lot of these have gone down, which again, if you're following crypto, you know there's a big crash recently, especially with with you know all these altcoins um that you're seeing right here went down a ton. So I think this is um Avalanche or die or whatever these ones are, you know, went down dramatically, whereas there's one up here, this Lone Wolf, um, that's just that's just did doing really well for whatever reason. So it's really interesting um to see. Now this is a pretty specific um visualization that I personally wanted to see, and I thought was interesting. You can do absolutely whatever you want to do with this data; I mean, there's so much here; you can do a lot—I mean, a lot—with this data, especially depending on how long you track it, right? I only did this over the course of like five minutes, but if you set this up um and you can track it over a longer time now um let's say you wanted to do something much simpler; uh, you just want to look at like Bitcoin over that time that you you know uh uh took the data in; that's going to be a lot simpler than what we just did, and I'll show you how to do that really quickly. So we're going to look at the data frame, and we are going to say uh or we're going to take specific columns; we just want um a few columns that we want to keep or or pull from. So we're going to take uh oops; we're going to take the name column; we're going to do uh might be easier if I copy them, but I'm just going to write them out: quote.USD.price; this is the price of the actual cryptocurrency; then we're going to do timestamp, and let's make this data frame, and we're just going to do 10 for absolutely no reason; maybe I should have made it nine; it would have been easier. So now we just have these um these columns, and you know we have all these separate columns, so what can do, and the re kind of the reason I want to show you this is you can just query this really quickly and just take the columns that you want. So let's say we just want to look at Bitcoin. So we're going to say data frame 10.query.open parenthesis, and we're going to say name is equal and equal is not like that uh when you're doing it like this; you need to say equal equal equal to oops, ignore that uh is equal to bitcoin, and we're going to do it just like that, and we're going to say data frame 10 is equal to. Let's try running that; I think something's wrong with it. Try like this. All right, let's try that. There we go; it was just the I needed a double quotation instead of a single quotation; that was the issue. So now we have Bitcoin; we have the price; and we have these.

Time stamps. So this is the actual time when we ran it. So this is the original data frame. And then in the, you know, this this project, it took me 15 more minutes to get this one, and then we had it running properly for the next five minutes. So that's, you know, that's actually what we have now.

If we want to just visualize this really simply, what we can do is we're going to say uh, we're going to do SNS.doline plot, and that's going to be like a little line chart or line graph, whatever whatever you want to call it. Then we're going to say x is equal to, we'll say quote, no, actually we wanted the timestamp to be on the x-axis, um, and then we'll do y is equal to quote. USD.price. Let's see if that works. Could not interpret timestamp for the parameter. Uh, that's because it's not understanding that data equals data frame 10. Now let's try this. All right, so this is uh, looks terrible. Let me me just say SNS.setor theme and open parentheses, we'll do style is equal to dark. This looks a little better.

Now, again, we are looking just at a a very very short time series, but we can look at just Bitcoin, or we can look at multiple, and we're showing this, you know, this line that's showing us this trajectory over time. So you can get really creative with this. You can run this for a long time; you can show Bitcoin over days, weeks, or months, however long you run this. And so that's really all I've got. Um, honestly, like I said, this is not a, I wouldn't say this is a complete full project, but I'm showing you how to do something to enable you to kind of run with it and run with the ball and do basically whatever you want with this. You can pull it from, you know, data from a different API; you can use this exact API and data, but I wanted to show you just a few things that I initially saw that I might do with the data. And you have so much. Let me go back to this original data frame. Uh, right, we use this one right here, this one right here. Look at all this data! I mean, you have so so so much data. Actually, let's go to this one; this one's better. You have so much data, so many numbers here, um, so many columns that we didn't even look at that you can use, um, and so you know, there's a lot that you can use here, and I'm really trying to just set you up so that you can run with it and do whatever you want. I could have done a thousand different things here, but you know, I tried to just show you two things that you can do with the data that I thought were pretty interesting or or simple to do, and you know, I want you guys to go out and do something way way better than what I did.

So I hope that this was helpful. I hope that this showed you how to automate that process so you don't have to sit there and click it and append it and do all these different things that it can show you how to kind of automate this process, and hopefully that will be helpful in your future projects. So with that being said, thank you so much for watching. If you made it all the way to the end, you guys are fantastic. If you like this video, be sure to like and subscribe below. I'll see you in the next [Music] video.

What's going on everybody? Welcome back to another video. Today I'm going to be walking you through how to create your very own portfolio [Music] website. Now we just completed our data analyst portfolio project Series where we walked through four projects in SQL, Tableau, and Python. And so if you have completed those projects, you now want to share them with potential employers, and I think the best way to do that is to create your own website. In just a little bit, I'm going to show you two options on how you can actually create your own website. The first one is a website builder like wix.com, and the second one is hosting your own website through something called GitHub Pages.

Now, if you have never created your own website before, it can sound a little bit daunting, but don't worry, I'm going to walk you through every single step of the way, from the very start to the very end. And once once you reach the end, you will have a complete data analyst portfolio website. So without further ado, let's jump on my screen and let's get started.

All right, so the website that you're looking at right now is the actual website that we are going to build in this video. Um, it is hosted on GitHub Pages or github.io. So this is actually being hosted right now by GitHub pages. So if you type this in—I'll leave a link in the description—if you type this in, um, you will get this page, and you can check it out for yourself if you don't want to just watch me look at it. Um, so you know, it has this little header, and you can write a little bit about yourself, and then these are our actual projects. So this is our data cleaning in SQL project, um, and then there's the Covid uh data exploration Tableau dashboards, movie correlation with Python, um, this is a future video. I plan on doing a few more of these projects because I just really enjoy them, so um, you know, and then there's this contact information at the bottom. So it's a really simple website, and it gets the point across, and uh, I have something similar to this from my own personal one. I I use a different variation, but um, this all comes from this website HTML 5 up there. Are lots of templates, lots of options that you can use. Um, again, the one we're going to be working with is this one, but I use a different one for mine, and they are really good. I'm in super easy to build and customize yourself. And I will say again, I have no experience doing this. I just swed a YouTube video that showed me how to do this, and now I am creating my own YouTube video to show you how to do this. So it's coming um pretty much full circle. So like I said, there's no no real narrative to it; it just clicks to your project. Um, if you click on this and let's just open it a new tab, it'll take you right to our to the GitHub project, um, and then you the this whoever is checking this out, like an employer or a recruiter, can see your code. So super simple.

Another way that you can do this is creating your own website through like a template or something like that, um, almost like a Blog style. So I imagine it being very something very similar to this, where there's this introduction, and you can talk about, you know, where you got the data set, how you got the data, um, and then you can kind of have a more narrative uh approach with screenshots and with some code as well. So you know, this person included screenshots, um, and then there's the code right here that I can actually copy um and paste that, and it just walks through the logic of how the project was done. Um, there's a story to it really, and so that might be something that you're interested in. Now I have done something like this in the past, and I used Wix, and there's a you can do this completely for free. Um, the one we're doing today is completely free as well, but you know, if you want the customized um the customized URL, you do have to pay for it on Wix, but you can get a free Wix website with the Wix um in the URL. So you know, try this out. These are super easy, and you can find thousands of templates and a million tutorials on how to do them. Um, so that's not the one we're going to be working on today.

So with that being said, uh, the very very first thing that we need to do before we do anything is actually download Visual Studio Code. This is where we're going to download that HTML, and we're going to be working with it in there. Um, again, I don't know if I said this before, but it seems a little bit intimidating at first, but once we actually start looking at it, it's a lot easier than it looks. I promise you. So if you are me and you have a Windows computer, you'll just go right here; you'll install it. Um, super easy to install. I'm not going to walk you through how to do that. Um, of course, I already have it up and running uh down here. So once you have that installed, what you're going to do is you're going to come to this website—a link should be in the description—we are going to download this. All you have to click is the free download; it's going to pop up. I'm going to put it in my downloads; I'm going to click save. Fantastic. Uh, so let's go to the downloads, and it should be right here. Now if we open this up, it has a few different things in it. Okay, so um, I'm using the Brave browser, so that's going to be right here. So that's just the symbol, but for you, if you're using Google Chrome, that should be the symbol there as well, but this is everything that you should be seeing, and what we want to do is we want to take it out of this um zip folder because there are things that can read into it with Visual Studio Code, but I want to make this as user-friendly as I possibly can. So what we're going to do is we're going to make create a new folder, and I'm just going to call it massively, or you can call it um Port website, whatever you want to call it. I'm just going to do Port website, um, and we are just going to—I'm going to copy this in; I'm not going to cut it in just in case I make a mistake—so going to put all of those um all of those things in here, and now what we're going to do is we're going to go to Visual Studio Code right here, and you should be greeted with this um this right here, and we're just going to click open folder, and we're going to go to Port website and we're going to go select folder, and you're going to say yes, I trust this one, and right over here is all of the documents that we were just looking at. Now the one that the only one really that we're going to be working in—we'll work a little bit in the images—because I'll show you how to add your own images—the really the only one we're going to be working in is this index. So again, it looks complicated um if you've never looked at HTML before, um it does look a little bit complicated, but HTML to me is one of the more easily understood languages um once you start kind of getting into it, which we're about to—we're going to walk through the entire process—it actually makes a lot of sense, and it is pretty simple. Um, something that you're going to want is you're going to want something called a live. So like if I click right here and I click open with live server—you don't have that yet, I'm guessing, unless you've done this before—it's going to open up this website, and this is what we're looking at right now. So it has a bunch of um gibberish or some language that I do not know, and so we can view this live um in just a second. I'm going to take myself off screen, but before I do that, um let's download or let's um search for that that live um I think it's called live share live server. Um let me see what this is called. Yeah, live server. So come right here; it's called this live server. There it is. Yeah, that's the one. So this is our live server. You just need to click install; it takes like five seconds, and it should be completely installed. Um, what this does is it just hosts a local website; it's not something that anybody can access, um, but it connects to your code, and when we make updates, it'll make a lot—you can see it live; you can see those updates live. So I'll show you all that in a second. Just be sure to um be sure to download that or install that. Uh, with that being said, let's get out of this; let's go let's go back right here. Uh, with that being said, I am going to take myself off screen so that you can see everything that I am seeing as well. Um, it's been really great seeing you. Have lots of different videos coming up, lots of new projects. Um, I just I really enjoy this project series. I think I'm just going to do more of them, so uh all right, I'm going to get myself off screen so let's look at what we actually need to do. So I'm going to um so let me see. Okay, so we're already connected to the live. Actually, I got rid of it. Whoops. Let's pull this over and let's pull that, and we're going to open and live server. So if we look right over here—and I know this is going to be a little bit squish, and I'm sorry about that—but if we look right over here, this says this is massively. So you can change that; that's that's this right here, and you can say we're going to say Alex the analyst portfolio, and we'll get rid of this massive. I'm going to hit control save; you can also go up here and hit save, but I'm I'm going hit controls. So I hit control s, and just like that, it updates on the website. Now, again, this is just a local, so it's nothing that anybody can see, so don't worry, but what we're going to do is I'm going to walk you through the entire process of creating this, and at the end, I will show you how to host it on GitHub, um, and it's honestly it's it's a fairly easy process; it just takes a little bit of time to customize it all. So let's get into it. So we have this—you may not be able to see it; let me actually pull this up—so it says massively by HTTP. We're going to customize that, customize that as well. Whoops, I don't want to do that every single time. I'm I'm going to try not to go full and go back and everything like that. So we're just going to say Alex the analyst portfolio, um ctrl s, and right up here that changed it. You may not have been able to see—yeah, don't ask me that again, thank you—right up here, you probably can't see at the moment; we'll see that later, but it it customizes this um tab, which is really cool. So let's go right down here. Now this is where it says a free fully responsive HTML 5 template. We can customize that, and I highly encourage that you do. So what you can do—and they actually included their Twitter handle right here—and you can do the same. If you look at this one right here, I included my Alex the analyst handle; that that goes to my YouTube channel, and you can do the exact same thing; include your LinkedIn or your GitHub profile or whatever you want to include in there, um, and so you know, be aware that you can do that. So let's say—oops, I need to click back in here—so we're going to say um data analyst skilled in and then the again, don't write what I'm writing; you can—it's—I'm just going to make it really simple, but you know, this part is meant to be a little bit about you, as who you are. So I'm going to say data analyst skilled in SQL, Tableau, and Python, and then I'm just going to get rid of all of this. Yep, yep, yep, everything from here over, and ctrl s. And so super simp simple. Actually, let me—where was that four, four here it is—we don't need that. Actually, we don't need anything from here over, probably here, honestly. See what that looks like, um, and yeah, and I can—again, you can use any website right here that you want, and you can customize what it looks like. So I'm going to say Alex the analyst, um, and then whatever URL you want to include in there, that's what you need to put. So now if I save—oops, oh, I I this is embarrassing; I am not a website—I am not a web developer, as you can see—but then if I go in here and I right-click and I say open link, it is going to take me to that project. So super simple, and we're going to do basically that for all of these. Um, I'm only going to show you three, and then you can do the rest, but I want to show you how to also do the um put the Tableau—it's the exact same thing, but you know, it's different, so wanted to show it to you. So the next one that we're going to do is go down to posts, and again, I'm going to get rid of this date. You can keep that in there if you want, excuse me, and that's totally fine; just update the date. Um, this is that

Said Magna again, I think this might be like some language I just don't know about. Um, the next one is data exploration in SQL, and I'm going to get rid of this, and we'll save that. Perfect. And we'll do view project. Cool. And yeah, so now we need to um customize this summary, and so I'm just going to say something really simple: um, data exploration of COVID-19 data set in SQL Server. There we go. Let's save that. We have view project. Now let's go get our project. So this is the data exploration; we're going to take this, we're going to copy it, and we're going to put it right in here and right in here as well. And if you want to, you can also include it right up here, so we have it in all three places. Uh, again, once you click on these, they will come up. Let's go to the next one. We're going to get rid of this. This one is going to be our Tableau projects. So actually, let me just copy that while we're here. This is going to be our Tableau projects.

So if you have one specific project that you want to include, what you would need to do is actually go in here, click view, grab that URL. What I am doing is I am just sharing my Tableau public page. So if you have tons of projects in here and um you want to display all of them, then or you want them to be able to see all of them and go and pick and see and choose what they want to look at, then just choose this URL that we're choosing right here. So um in here, on in the um HTML, we're going to put—I'm going to put Tableau projects—and let's go like this, and then we will get rid of uh that hashtag, pound sign, whatever you want to call it, and we'll hit Ctrl+S. And oh, we got to do the um this as well. This is my—this is going to be a terrible, don't use this—this is my Tableau; this holds—I'm just—this is bad—this holds all of my Tableau dashboards. Don't, please don't do this. Um, I am doing this because I don't want to take forever in a video to make it perfect. Um, and then you know you're going to do the exact same thing. So in this one right here, I included four, so I'm going to keep four. Um, let me do the uh—no, I'm just going to do these three; I'm not going to take up more of our time. Um, so we did those; I'm just going to keep these three in for visual purposes. But once you get down here, um you know what we're going to do is delete some of this right, so we—this is our data exploration, and where's our Tableau? This is our Tableau right here. So Tableau projects; they're separated by these articles. So what we're going to do is go around right here and we're going to go down, down, down, down to right here. This is going to get rid of all these other articles or all these other what they're calling um um posts. So we're going to get rid of those, and we're going to hit save. And now, as you can see, we have our header, we have our first project, and we have our second and our third. I would include those other projects that we've done in here so that it looks good. This is this footer right here; we don't need that because we don't have any um anything else in there, so we're going to get rid of that as well. And now we just have this information. Now I don't have anything where they can do the name, email, message, or you can keep that in there if you'd like, um but I am going to get rid of this. So we're going to go right here; that's the section, so don't delete the section; we want that. I'm going to delete this footer section is what they're calling it, and now we have this address, phone, email, social. Um, and I'm going to get to the Social in just a second; it's again super easy. But for the address, I just put location; I don't want to give somebody my address or put it on a website anywhere. Um, it's not something I want to do. So what we're going to do is just put—I'm going to put Dallas and Texas—and we can keep it like that, and we'll hit—oops—we'll hit save, and it'll have Dallas, Texas. Um, hate the look of the zeros, 6s, 89 Z. So we're going to do that phone number: 1 2 3 5 6 7 8 9 0. And then email, and we'll put AlexTheAnalyst95@gmail.com. If you have issues with this, um you can email me, but I'll try—I will try to respond to all your emails. I get a lot, um so I will do my best. That is my actual email if you are curious.

Now, um now that we have this, we also have these—this social media. Now I want to display my LinkedIn, and I also want to display my GitHub. So what I'm going to do right here is I'm going to go over here and do LinkedIn. Perfect. Let's go to this. So I'm going to take my LinkedIn URL, and I am going to get rid of these first two because I'm only going to include two. And for this one, I'm going to do uh LinkedIn, o LinkedIn. And then for right here, I'm going to replace that with LinkedIn. And what you're going to do is put this link right here, and then we're going to go get to get the GitHub. So let's do GitHub. Oh, who is this sign up? What is going on? Um, I don't—there—let's just go back here. I—that was some—I was like viewing a while back or or something. Um, so we're going to take the GitHub and we're going to put that right here. So it already has it as—um, the GitHub—is this supposed to be lowercase? I think it is. Let me see if this is lowercased as well. Yeah. Um, so do it like that, do it lowercased. Um, I forgot that that was how they did it. Um, and oh, that's the label; that doesn't matter as much. But this right here is the class is actually the important part because then when we go back here, there is no LinkedIn image, but when we save it—oops—when we save it, it has the LinkedIn image because it's already a class that was created in this HTML um template. So we have that. Um, and let me bring this full screen really quick because there are a few things that we couldn't see in that that screen. These right here are things that we could not see before, um and these as well. So what we can do is we're going to go down here; we're just going to copy these social; we're going to replace them right here so they can have those, and then we're going to get rid of these two right here. And this says this is massively, and we're going to change that as well. Let's make this full screen for the first time; feels good. Um, I hate doing split screen, but I do it for you guys. Um, so this is massively, and we're just going to put—we're just going to get rid of these two. This is—it's called The Navigator, the the different tabs—we're going to get rid of those two tabs, and then for this I'm just going to call it projects, and I'll—once I—once we go back and update all of this, then you will—you'll see those changes. So let's see. So we made those changes. Here's our social or the social medias—Social Media stuff—we're going to go and copy these two and we're going to replace all of these with this, um and let's save that and let's go back. So now, as you can see, those two are gone; this says projects; there's only two right here, and if you click on it, it's going to go to my LinkedIn or your LinkedIn when you do it, um and this will take you to the GitHub. So it is all working as intended; this is great. Um, when you scroll down and it says massively, we can change that as well, and we should. Let's do that really quick. Um, we'll just say Alex the Analyst, and we'll update that, and there we go.

So in a nutshell, this is a lot of it. Um, we need images, and I don't think I set this up for this video, so I'm going to—I'm going to like cut myself off for like two seconds, go pull those images in, um because it could take like a few minutes; I don't want to waste your time. And then—and I'll come back. So I'll see you in two seconds. All right, so I just pulled over the images that we are going to use. Let's go to the downloads; um, they're right here; they—the housing, Tableau, and Co—um, if I open up this Co one, this is what the image looks like; this is what we're going to use for that COVID project. So I'm going to copy these; I'm going to go into the port website um that we just had; I'm going to go to images, and I'm going to insert these in here. So now that we have those images in here, let's go back and let's see what we got. So we just put these images in—this um you'll have this folder right here, and you can open it up, and you can see all of these that we have. So all we're going to do is go and replace the images—these these, you know, um temporary images that they had for us—and we should be golden. And then we're going to actually upload it to to GitHub and then create our website for free. So let's go right down here; this is our very first uh one; this is our data cleaning in SQL; this is with the housing data. So this image right over here, it says images1.jpg, so jpeg, I don't know why I said it like that. So this is the housing. So what we're going to do right here is do housing, and it'll autocomplete for us, um so that housing should be in there now. Next one is the data exploration in SQL; that was with the COVID. So we're going to get rid of this; I want to say COVID, um because that is the image that I have right over here. And then the last one is—excuse me—Tableau. So let's go right over here; let's do TBL—low—let's get rid—oh, I got to save that—Ctrl+S. Perfect. And now let's look at it. There you go. There you go. Oh, this one still says full story; go change that. Um, I'm going to go change it; it just doesn't feel right. Uh, View project—oh, that's not how you sp—okay, Ctrl+S. Perfect. Okay, so now this looks a lot better, um and when we host it um through GitHub Pages or github.io, this is going to be what it looks like. I mean, it is, and you can add a lot more to it; you can take away from it; you can add as many projects as you want; you can keep adding; you can copy those articles or those posts, and you can just keep adding them. Um, so this is kind of what it's going to look like, and it was not that hard; I don't think. I hope this was not too difficult; I really don't think it is. Um, it's really just using a template and kind of understanding a little basics of HTML. So um we are going to take this, and we—we have this saved already; we have this all saved. What we are going to do now is upload this to GitHub. So let's go right over here; let's go to here, and let's go to repositories, and how do—where—where's the new one? Oh, I need to sign in. Okay, I'm going to get rid of this part so you can't see it. So we are going to say a new repository; we're going to call it AlexTheAnalyst2.github.io. So we're going to write it just like that. You know, if your name's um Alex Jimmy—I don't know why I said Jimmy—Alex Jimmy, Alex jimmy.github.io; you can always go back after the fact and change this, so it's not a big deal whether you change it or not. And we're going to create this repository; we're going to say upload an existing file, and instead of choosing them, what we're going to do is just go right over here, go to this, and we're just going to copy this in or not copy it in but drag it in. Okay. Okay, so we're going to take this, drag it in right here, and it can take a—it'll take a little bit; has a 75, but it shouldn't take that long. And let's just wait for it. I'm taking a sip of water; I apologize, but it is literally uploading just everything that we had in there, so all the updates and all the changes and all the stuff that we um had, and it looks like it's done. So let's just write initial commit, commit changes. It is processing it. All right, and it should be done very, very soon as long as I have a good internet connection; we shall see. Stick with me; it's taking its time. Um, while it's loading, let's go over to—oh, oh, there it is. So perfect. So here's everything that we have; has this README that it generated. Let's over to settings, and we have this—github.io. And if we go right down here to GitHub Pages, pages settings now has its own dedicated tab. Let's check it out here. So it is—it's currently disabled, but we're going to say we want it to do pull from the main—I think it's the doc; we'll see. I'm going to save this. Your site is ready to be published. Let's open this up. Okay, site not found. Maybe it's from the root. Save. Um, your site is having a build a problem. Let me see if I can actually change the name; I already have an AlexTheAnalyst, but I'm gonna see—it's already taken. Um, I'm just going to try this one one more time. Oh, and now it's working. Uh, I have no idea why it uh didn't work before, but this is fantastic. It was giving me all this—I was—maybe I was just reading too much into that. I had—I had never tried to create another um .io or or GitHub pages on this. So anyways, thanks for sticking with me through all that um stuff. So now we have our actual website. Um, it doesn't look the same up here because of that thing that we were just looking at; it should just be this part right here, but um this is an actual website now; it's being hosted through GitHub, and it's completely free. If you want to pay, you can hide this from your GitHub—your repository has to be public—something I didn't mention when you're doing this: your repository has to be public. Um, if I change the visibility to private, um you will not be able to see it anymore; you'll have to then pay. If you want to make this repository private, you have to then pay; I think it's like $4 a month or something like that. So worth looking into um if you don't want to display that on your GitHub. Worth looking into. But this is our final product; I mean, it looks pretty fantastic, and you can use any of these templates right there; are lots of different templates that are fantastic; I mean, they look amazing; they look professional. Um, it's really up to your style. Like this one looks kind of cool, a little bit um edgy for for my taste, but uh this one looks really good too; might—might be able to add some more narrative to that one. So again, go through it, make your—make a good choice in it, and then update it how we updated it. Uh, I will include the—let's see—I will include everything that's in here, and I'll keep this on my—on this GitHub so that you can go in there, and if you want to download these images, you can download the images that I used, um or you can go find your own; just um, you know, look for—try to get like HD images on Google; just type in Google images and search for whatever image you want to search; try to get an HD image. With that being said, that is the entire project. I—I—I—I hope this didn't go too long. Um, this may have gone—you know, this may have gone like 30, 45 minutes, but in the end of it—at the at the end, which is where we are now—we have an entire website; it was completely free, and I hope that you can now host the projects, and you can create—create more projects. I will be coming out with more projects myself that hopefully will be interesting to you in the future. So with that being said, thank you guys for joining me. For you who stuck it out to the very end, you are fantastic. You know, send me a post to your website on LinkedIn and tag me in it because I love seeing um you guys do these projects and this stuff. So I'm super excited to see all of these um that you guys tag me on on LinkedIn and whatnot. So with that being said, this is it. I hope you learned something; I hope that it worked for you, and I appreciate you watching. Be sure to like and subscribe below, and I will see you in the next video. [Music] Goodbye.

What's going on, everybody? Welcome back to another video. Today I'm going to help you create a data analyst [Music] resume. Now, when I say data analyst resume, it's not that much different than a regular resume, except that it's going to be catered for a data analyst job. In just a second, we're going to take a look on my screen at a sample resume. I'll have the template in the description, so you can just go and download it and fill in your information, but it's a fantastic starting place to actually creating your resume. When we're looking at this resume, we'll take a look at each section, kind of dissect each part of it, and then at the very end, I'll give some extra tips on what you should include and how to actually write your resume as well. So without further ado, let's jump onto my screen, take a look at the resume, and see how you can create your own data analyst resume.

So here's our simple resume. I'm just going to walk through the entire thing super quick, and then we'll break down each section individually. I'll give my thoughts and some tips on each section, and remember you can download this exact thing in the description below. I'll have a link; I'll probably put it on my GitHub or somewhere else, but it'll be free to download, uh, so you can go ahead and do that. But let's zoom in just a little bit. So at the very top, we have our header; we have some just basic uh contact information; then we have skills; then we have projects; and notice the projects are up here at the the top, and we'll get to that later about the order of where you should be putting your things; then we have work experience, and then we have education. So really quickly, I'm going to zoom out, and I hope you can still see it. The order is actually quite important. Now, there is one piece that is not in here right now, and that is a summary section. I don't have a summary section on my real resume; I just—I don't think it's useful or helpful; I don't have one. You can include one, and it would be right up here at the very top. Now, why do we have the skills and projects at the top? Well, it's because that most people who are trying to break into data analytics don't have any experience in data analytics. If I am reading this resume as a hiring manager, and the first thing that I look up here and I see is experience, and it's not analyst, it's a teacher or a nurse or something, I'm going to be like, this person doesn't have any experience; I don't want to hire them. The first thing that you want to have in your resume is something that is good for the hiring manager to see. The first several things you should put—all your best stuff at the top—that's my uh what I believe. So I think that these skills are really strong; a lot of great skills; and then these projects are all really good projects. Now, this is just a sample; these aren't all real projects, um, or they are real real projects; they're just not, you know, ones that I built myself; it's just a sample. So uh, then right here we have our work experience. Now, if you're, like I said, a nurse or a teacher or a lawyer or something that's not relevant to data analytics, you want that…

At the bottom, um, and then you're going to want to tie in uh some things in these descriptions, and then the education at the bottom. My education was terrible. Okay, I had a bachelor's in recreational therapy, which had nothing to do with data analytics, so for a tech job, it was not good. I always had mine at the bottom, so let's start at the very top and walk through each section.

So at the very top, you want to have maybe a title, but for sure your full name. You definitely want to include your phone number if you're okay with them calling you, but definitely an email. For sure, include things like a LinkedIn profile or a GitHub profile. You can also put your portfolio; in fact, I highly recommend putting your portfolio because it just looks good, or if they check it out, that's a really good thing. And then your location, cuz sometimes your job is going to be location-based, whether you're in Dallas or another Metropolitan City, it's just nice to have that on there. This should be the simplest one to fill out unless you haven't built out something like a portfolio; you just don't include it, um, but this one should be the simplest one, right? You're just putting contact information, maybe a link to a website.

Next, we have the skill section, and this one, on my own personal resume, I have at the very top. I typically recommend anyone who does not have experience who's trying to break into data analytics to put this at the top as well and have these skills and know these skills; that's important. Um, but when the hiring manager first initially sees this, it's just going to be a mental check: okay, they have the skills that we're looking for; let's move on to the rest of the resume. Um, but you want as many mental checks for what they're looking for at the beginning; just going to—I'm going to keep repeating that. Um, this is how I personally write my skills, so I write something like SQL, and then I'll say SQL Server, MySQL, PostgreSQL. Now, I have used all these different types of SQL in my actual job. If you don't—you haven't done that and you're just starting out, maybe you put something like, um, you know, subqueries, store procedures, joins, whatever the actual things within SQL. I don't really think—I don't recommend that as much because typically people know what SQL is like; if they use SQL, they know what SQL is, so they're just going to expect that you know those things.

Now, for something like Python, it's different because there are packages or something. R, there are packages and libraries within them, so you can specify: I have worked with pandas in my actual job, and I look for people who know pandas as well because, you know, we use it. So actually specifying these packages or libraries is really helpful. So this is how I would put these things on a resume. Now, this is another resume; this is our sample two. I'm going to maybe include this one down below, although I don't like this format as much, but if you like it, you can, but here's another way that you can um show these skills, just a different way to do it. I want to show you both ways um where we have like Python and the libraries underneath it. I've even seen it to where people write out almost like—um, let me go down here—they'll write out like a narrative; they'll do Python, and then they'll have like a colon, and then they'll say: use to um manipulate data in pandas… and they've write it out. You can do that as well. Again, I'd like bullet points because it's to the point; it's exactly what you need. Let's get rid of this one real quick. So this is the one uh that I like, so that's the skill section. Let's move down to the projects.

Now, the project section is almost primarily for people who are just starting out. Once you get experience, typically you maybe have one project on there or no projects at all, but the project section is used as kind of um, an in lieu of actual experience, right? I've always said that you need to build projects not just for your resume but also for the interviews, so then when you get into an interview, you can point to these projects and say: yes, I've used SQL; I did it in this project, and they may have seen it, and you can walk them through how you actually used it. It gives you more credibility than just saying you know how to use SQL. So within the project section, we're going to have a project like this one says: Data Science Job Market Exploratory Data Analysis. So this is a personal project, and then within it, they did some really great stuff. Here's usually what I recommend, and this is in here, which is: you specify what you did; you say: I used Python, and what did you do to analyze this and gain insights in the job market? Then you walk through some of the things that you actually did, things like regex techniques you used, pandas, matplotlib; you built a wordcloud. These are keywords that somebody will look for, and they even highlighted them, which I personally like and do as myself. They highlighted these things so that the viewer or the um hiring manager is actually seeing them, making sure that they're bold so that they are catching their eye. So I personally do this, and I recommend this; that's all it needs to be; it just needs to be: I built a Tableau dashboard doing this from this data set; I cleaned it in SQL, and you show those skills. Something that's important in both the skill section and the project section is using and highlighting your skills as much as possible, especially if you don't have any experience. If you've never had a job before, once you have a job and you come down to like the work experience, then it kind of speaks for you, but if you don't, you want the projects and the skills to speak towards your skills and credibility. So we have this right here. Now, one thing that's not in here that I actually do recommend is a hyperlink, maybe right here, or actually this being a hyperlink to the project because they might read this and be like: I—we work with, you know, data science job market data; I don't know, and then they'll click on this link, and they can see your work. That is the one thing that I would change in this, other than that, this is exactly how I would have it, very, very, very similar to my own, um, and a lot of this that I did, I actually took from other resumes and formatted how I prefer and like it, um, so again, some of this is personal preference, and you can change it however you want; that's just how I like it. So that is the project section. Now we're going to go down to the work experience section.

Now, this person does have a little bit of analyst uh experience, so you know, if you don't, that's okay, but you put your previous experience. Now, here's what I recommend: if you've been a teacher for 15 years, you've been a nurse for 10 years, you've had 10 different jobs, don't put all your experience on here; um, maybe put your last two jobs, going back maybe three years. I don't recommend you filling it up because it's not going to be super relevant unless you're applying for a healthcare data analyst position and you have a nursing degree, then it's relevant, and that experience is super helpful because it's domain experience, right? Then you may go back five years; just, you know, use your discretion, but what do you need to include? Of course, your title, where you worked, your location, and the times; that stands for almost any resume, but within here, uh, what you really want to do is highlight again the skills, if you can. If you can't, that'll change, but in here he says: implemented a new reporting using Excel pivot and VBA, which reduced processing time by 50%. These types of um quantitative information—I reduced time, I—I saved the company money, I—I did something quantitative—putting that in here is always helpful, always highly recommended, although it can be tough to measure these things, right? Typically what I recommend, especially if you're first starting out, is to highlight skills. If you're a teacher, you've probably used Excel, and you've probably used Excel for closer to data analytics than you think, just in a teacher way and not a data analytics way, but you can reword these things and make them sound good. If you are a a nurse, like I was saying, you've used Excel, you've used a health information system, you've used uh some type of database; talk to that; include that in here, um, and it can be hard to write these out, and I'm going to show you a way in just a little bit about how you can write these out and think about these things or have a way to help you write them or give you ideas. We'll get to that in a second.

Lastly, we have the education piece. This is again really simple: at the very bottom, education, what your degree was, where you went, um, and if you have, you know, some helpful things to include, you can do that, and then when you actually went. Now, you can include other things in here as well, like boot camps, if you went to a boot camp, or you could also include things like a GPA, although I don't personally recommend it. GPA has never been anything that I've ever cared about or I've seen anyone care about ever, um, so you don't normally have to include it. One other thing that you can include at the very bottom is something like certifications. Uh, I personally don't put a lot of stock in certifications unless it is one that I have recommended in previous videos, like the Tableau certification or Tableau Desktop certification. If you're applying to a job that uses Tableau, that actually could be really good, so definitely include that, but ones on Udemy, ones on Coursera, or like my Alex the Analyst boot camp that I have on my channel, I wouldn't really include that in your resume; it's mostly for learning. If you get something like the Tableau one or the AWS uh Cloud one or the um Azure Cloud one, those are all actual certifications that can help you and give you credibility towards a certain skill.

Now, really quickly, let's just take a glance at the other resume; this is resume 2. So we have the education at the top; doesn't have to be at the top unless it's relevant, which you could put at the top. We have a skill section; they—again, this is the project, same projects, and then work experience; this is just a little bit different um order, so you can do it like this as well, in a different way; you can write the skills, and you can also include a summary section as well. So that's the meat and potatoes of how I would create a data analyst resume. Now, writing it is actually a different beast, right? You have to actually write it out, get something on the resume, and then apply using that resume, but it can be hard to come up with these ideas, so uh, I just want to show you something that a lot of people have been using; I personally haven't written a resume in a little while, so I don't use it for my own resume or haven't used it, but I will, um, and that's using ChatGPT or some variation, whether it's on Bing or, you know, you get some different version or some new product that's out there at the moment. I'm just going to show you how to do it in ChatGPT; some of the things that you can prompt it to do, and that'll be it. I'm just going to show you kind of some ideas that it can generate for you to help you write these things. All right, so here in my screen, we're on ChatGPT. If you haven't used it, I'll leave a link in the description. I also have a whole video on how to use ChatGPT for data analysis, um, so I like ChatGPT. Now, I've already written out these questions because I don't want to wait for the responses, but here's what I asked it to do, and you can do some variation of this whether you're a nurse or a lawyer or a teacher, whatever. I said: I'm a math high school teacher trying to become a data analyst; how can I use my experience on my resume to help me get a job? This is just to help provoke some ideas, and it says, you know, you most likely have some skills; emphasize your quantitative skills, so those are some of the things you can focus on; showcase your ability to communicate complex concepts, which is really important in data analytics; being able to present information, which teachers have; highlight your experience with technology; hopefully you're using some type of uh, you know, database for students or, you know, Excel or something like that, and you can highlight that and showcase your ability to solve problems. Now, the next thing that I asked it was: I built a COVID Tableau dashboard using Tableau; how can I add this to my resume? And then it's going to tell you exactly how you can do that; it's going to say: include the link to your dashboard, which I also recommend; provide a brief description; highlight your data visualization skills; include screenshots or images, which that's what I would be putting in the project itself, not on your resume; then provide context for the data; all really good stuff, really great. Now, the last thing is kind of what I'm trying to get at as a whole; it can help you write things, so I'm going to say: write a two—I said: write a two—write two sentences highlighting my COVID Tableau dashboard to add to my resume, and it's going to say: developed a COVID Tableau dashboard to visualize pandemic trends using real-time data sources, demonstrating strong data visualization and analysis skills. So this can help you generate those descriptions in your work experience; it can help you generate the descriptions in your projects, and this can be really helpful to just generate some ideas, cuz I personally really struggle with like highlighting my skills and descriptions within those things; this can be a way to kind of help you do that, so don't, you know, just copy and paste, but let it prompt you; let it give you ideas.

Now, the last thing that I want to mention is just your overall resume as a whole. The template that I use, the template that I recommend, is very, very friendly to these automated systems that check your resume. If you did not know, most companies, especially big companies, use these automated systems that scan your resume, see if it has what they're looking for, and then that resume, if it gets through that system, gets passed on to a recruiter or hiring manager. Typically, most companies don't go straight to the hiring manager, so you need a resume that can pass through those initial systems and pass those tests. The resumes that I've shown you today will do that; they have bullet points; they have the keywords; they have everything you need; that's why I recommend, or partially why I recommend, this type of resume. Other ones that have images and different fonts and different stylings can cause issues with these automated systems where it just doesn't read it properly or, you know, it doesn't read the right words that you want it to read. So just know that these types of resumes have different uses, right? You're not just handing it off to somebody to where they can read it, and it's needs to be visually stimulating; really what you need is you need it to get through those initial systems, which these resumes uh, if you write them well, you have good, you know, skills and the right things on your resume, they will pass—pass through that first layer to get to those hiring managers. So again, be sure to download those; those are completely free; I just—I highly recommend using them; I think they're really good. So be sure to download those, use those, just put in your own information; be sure to build out your own projects; don't just keep the ones that are on there because you'll need to be able to speak to them; sometimes recruiters or hiring managers are going to ask you about them, how you built it, what you did, and you can also point to those projects in your actual interview. So I hope that this was helpful; I hope that your resume is ready to go; I hope that you're ready to start applying for those data analyst jobs. Thank you guys so much for watching; I really appreciate it. If you like this video, be sure to like and subscribe below, and I'll see you in the next [Music] video.

What's going on everybody? Welcome back to another video. Today we are going to be solving easy SQL technical interview questions. [Music] Now, when I was interviewing for data analyst positions, I almost always got some type of technical interview, and the vast majority of the technical interviews that I got were in SQL. Not only that, but I was also on a hiring team, and then later as a hiring manager, I almost always conducted some type of SQL technical interview. Now, why do hiring managers conduct these interviews? It's because they want to make sure that you actually know the skill because you can just put SQL on your resume and not actually know it at all, and then when they hire you, they have to spend two, three, four months training you on the basics of SQL for you to actually understand it and use it. That is not what any hiring manager wants, and so that's why they conduct these SQL technical interviews. So in this series, we're going to start with easy questions that'll be in today's video, then we'll go on to medium, hard, and then very hard SQL technical interview questions. So without further ado, let's jump on my screen and take a look at the easy SQL interview questions. We're going to be practicing these questions on analystbuilder.com, and if we come right over here, we can filter to the free questions, and we can filter to the easy questions. These are all the free and easy questions that you can go and take right now. I will leave a link in the description; I will also leave a link to the two questions that we're going to be looking at today, but go ahead and check out this if you want to try out all of these different questions; there's also different difficulties. So we're going to be working through the moderate and the hard ones in future videos, and then we'll also be looking at the very difficult ones in the very last video in this series, but let's go ahead and get rid of this cuz we're going to go over to our very first question. Now, really quickly, I just want to show you the interface before we actually dive into the question. Now, we're going to be solving this in MySQL, but you can also practice in PostgreSQL and Microsoft SQL Server; whichever one you have an interview coming up for, if that company uses Microsoft SQL Server, come in here and use Microsoft SQL Server, and we also have Python as well, but we're going to be doing this in MySQL. This is where we'll write our actual SQL code; then we have the prompt. Now, in an interview, typically they're going to give you some type of prompt, and then the data; they're going to ask you to do something with the data, and our data is right down here. So this is our data. Now, this is a practicing platform, right, to practice for technical interview questions or practice for technical interviews, so we also have hints and expected output as well as a video explanation walking through this question, showing you exactly how to do it, um, but if you can't get it at all, you can always come in here and look at the solution for any of these, but let's go ahead and start this question. The question is called Car Failure. It says: Cars need to be inspected every year in order to pass inspection and be street legal. If a car has any critical issues, it will fail inspection, or if it has more than three minor issues, it will also fail. Write a query to identify the cars that passed inspection. Output should include the owner name and vehicle name, ordered by the owner name alphabetically. Let's go take a look at this data. So we have the owner name, vehicle name, minor issues, critical issues, and that's all we have, so it's just a simple one, um, again, this is an easy question, at least on Analyst Builder, and so let's try to solve this without any of the hints or looking at the expected output because I think we can solve this one. Now, the first thing that we need to take into consideration is we're going to need to be filtering, so we'll need to filter down some data. So if a car has any critical issues, it'll fail, so if critical issues is 1, 2, 3, it's going to fail, so it can't have uh no critical issues, and they'll

What's going on everybody? Welcome back to another video. Today we're going to be solving medium SQL technical interview questions. Now, if you watched the first video in this series, you saw that we answered some easy SQL technical interview questions. Now we're going to be solving medium-level questions. The medium questions are going to be a little bit more difficult, but I will say that if you are practicing for a SQL technical interview, I highly recommend practicing the easy and the medium questions. But let's not waste any time; let's head over to my screen and take a look at our medium-level questions.

But actually, before we take a look at our two questions up here, you can come over here, go to the free questions, go to the difficulty, and go to moderate. These are all the medium-level questions that you can try for free on analystbuilder.com. If you have not tried out Analyst Builder, I highly, highly, highly recommend it. That's my data analytics learning platform that I'm extremely proud of. You can take my full courses and try out these technical interview questions all in one place. But let's head over to our first question, which is called Tech Layoffs.

It says, "Tech companies have been laying off employees after a large surge of hires in the past few years. Write a query to determine the percentage of employees that were laid off from each company. Output should include the company and the percentage to two decimal places of laid-off employees. Order by company name alphabetically." And I think this is extremely accurate because that just happened—I'm recording this in late, late, late 2023—but I'm sure I'll release this in 2024, and I think you guys know what I'm talking about. It was just a bad year for 2022, 2023, all the layoffs. Now we have Apple, Microsoft, Google, Amazon, Facebook, Tesla, and these are the employees fired, and what we're trying to do is output a percentage. So we need to look at the company and then the percentage of laid-off employees. If they had zero laid-off employees, the percentage should be zero. But let's say they had 6,000 of 181,000; we need to see what percentage of the entire company size was laid off—was it 1%, 2%? That's what we're trying to determine. Now we always can use hints and expected output if we need help or if we need the video walkthrough, we can use it, but I don't think we'll need it. Let's come over here and make some notes before we get started.

Now, before we write anything, remember when you're in a technical interview—whether it's for data analysis, data engineering, data science—it doesn't matter; when you are applying for these jobs, writing it out correctly is important, but I would say even more importantly, it's about how you actually talk through the problem because that really shows your skill level. If you're walking through it and you're just typing random stuff and it kind of makes sense, but you're not talking out loud, they may not understand that you really know what you're talking about. And so you want to practice these questions, know what you're talking about, and while you're solving them, do what I'm about to do, which is I'm going to kind of talk through the steps that I need to do, then I'm going to write it out. That's what I recommend during actual interviews. So the first thing that we need is we need to find the percentage. Now this is going to be a calculation, so I'll write "percentage calculation." Now, how do we determine what the percentage is? What we need to do is employees fired divided by the company size times 100. So it's employees fired divided by comp size times 100. That's the calculation that we need, so we are going to go and do that in just a little bit. But after that, our output needs something, so our output needs—and I need to comment this out—our output needs the company name and it needs the percentage. So we're definitely going to need to include both of those. And then lastly, we need to order by the company name ASC, which means ascending, so A to Z. So this is what we need to do. Now let's do one thing first: let's just pull this up, but we don't really need to start looking at the output or the order by just yet. Let's keep everything; we'll keep this comma here; let's keep everything, but let's start working on our calculation. So let's see if our calculation is correct. It should be employees fired divided by the company size. I'm going to put this all in parentheses just to make sure we're doing PEMDAS correctly, and let's multiply it times 100. So let's run this, and here we go. Now this looks correct, just glancing at this, because here we have 0% because Apple didn't lay anyone off; 3%, 6,000 of 1800; that also looks correct. This one I think is the most straightforward one: 15,000 into for 140,000; that should be around 10%, but because it's 15,000 to 140,000, it's a little more than 10%, and so this one looks very right to me. Now, one thing I didn't mention here is we need to round to two decimal places. So let's go ahead and round this before anything. So let's put ROUND and let's wrap this entire thing. Now we need to round this to two decimal places, so we need to do is do a comma two here because that says rounded to two decimal places. So let's run this, and there we go. That all looks correct. Now we can say this as—let's just rename this as percentage—just so we don't have that really long name; it basically makes this the column name that's way too long, so we're going to call that percentage. Now the only thing that we need on our output is company name and percentage, so let's come back here and let's put COMPANY and let's run this, and this looks good, except we need to order by the company name ascending. So we'll say ORDER BY, we'll do COMPANY, and then we can say ASCENDING, although by default ORDER BY is in ascending; we just—I like explicitly writing it—so let's run this, and then we have Amazon, Apple, Facebook, Google, Microsoft, Tesla. So this looks great. I think this is our final answer. Let's go ahead and check our solution, and there we go. We got the solution correct.

Now, if you remember in the last video, we checked our profile; we earn points towards our badges. A medium question, you earn 25 points, and so if you're following along, you're doing these questions, then you should go check your profile out because you should have—in your profile, you should have more points. Now let's go to the next question. This one is called Separation. It says, "Data was input incorrectly into a database. The ID was combined with the first name. Write a query to separate the ID and first name into two separate columns. Each ID is five characters long." All right, I've seen this in real databases a million times, usually when we're like getting data from like an Excel file or something; we always have issues with Excel or CSV files or stuff like that. So let's talk about how we are going to actually solve this. Now we're doing this in MySQL, but again, you can do this in Python, PostgreSQL, Microsoft SQL Server—whichever one you are practicing or, you know, you have an interview coming up—whichever one you have an interview coming up, go ahead and use that one. Now I think one of the main things that I'm interested in right here is that each ID is five characters long because we need to separate this out. So it doesn't matter how long this name is; what matters—that's kind of key to this—is how long this ID is because if it was 3, 4, 5, 6, we might have to use something like regular expression to separate that out to extract all the numbers. But luckily, it's all five characters. So with this, we should be able to use something like SUBSTRING; this will be to pull out numbers and then names. So that's what we need, and then we'll have two separate columns. So the output will then be the ID and the first name, and I believe that's all we need to do is separate them out into two separate columns, and then we needed to figure out how to actually separate it. So I think that's all we need. Let's pull this up. There we go. Now I'm going to keep everything, so just keeping this column, but then we'll separate it out into two, and we'll see what this looks like. Now this SUBSTRING is going to take a few different parameters. First, we need to pass through the string. Now, when I say string, I mean the column that contains the string because it's going to go through each row of that data. We need to select the ID, and then we need to specify the start position and the end position. Now that's where this comes into place, where that five characters long comes into place. So because it's five characters long, we should start at position one, and then we'll do a comma and end at position five. Start at position one and take through position five. Let's just run this and see if it works. There we go. So this pulled out just the first five characters in this string. Now we also need to pull out the full first name, and we can actually label this—let me bring this down—we'll do AS—let's name it something; I'll say NEW_ID; that's what we'll name it, and then we'll do the next one. So this will be SUBSTRING. Now we also are going to pass through the ID, but this time we're not starting at position one. Now we need to start at position six, but we don't know how many characters are in each name. It could be S; it could be Harry; it could be—uh—Escariot; I don't even know if that's a name, but it could be really long—Alexander; that's my name—long name. So we don't know how long it could be, so we could put something like 20 here, and if we run this, it's going to extract it, but what if someone's name is longer than 20 characters? That's not good. Luckily, though, that third parameter, the end position, we can just leave blank, and it'll go from the sixth position to the very end of that string. So let's run this, and that looks really good. So we're going to say AS first_name, and there we go. So let's run this again. So we have the new ID and we have the first name. We can get rid of this initial one that has everything, and I believe this should be our full output. Now we have this new ID and we have this first name. Let's go ahead and check our answer, and there we go. Your solution is correct.

Now these are just two of the medium questions on the platform; there are a ton of others. So I'm going to leave links in the description to these questions as well as just to the questions page so you can go on there, and you can practice, and you can really get comfortable writing these out and using them because when you feel comfortable going into that SQL technical interview, you're going to do a lot better, and you'll know how to talk through these things. And if you're ever having trouble with them, you can always go to this video explanation where I will walk through it and tell you my exact thought process on how I solve these questions. And honestly, I think that's one of the best features about the whole platform because when I was first starting out, I didn't have this, and so I'm really happy that this is here for you so you can really learn. It's really just a learning platform to get better at these skills. So if you have a technical interview coming up, either in Python or SQL, try out analystbuilder.com; it is phenomenal. I created all the content myself. We

Also, have a full course on there, so you can go ahead and check that out as well. In the next video, we're going to be going onto the hard questions; that's going to be quite a big leap from medium to hard. And then after that, we're also going to be going into very hard questions. I would say the very hard questions are more of like a challenge; they're very difficult, they're a lot of fun, but go ahead and check all that out on Analyst Builder. With that being said, if you like this video, be sure to like and subscribe, and I will see you in the next video. [Music]

What's going on, everybody? Welcome back to another video. Today we are going to be solving hard SQL technical interview [Music] questions. These hard interview questions are something that you would get in kind of a medium or a senior-level data analyst position. The easy and the medium are more towards the entry-level or SL midlevel, somewhere in that range. The hard ones are not something that you're going to get in the kind of more entry-level range; these are questions that you might see in kind of a more advanced SQL technical interview. I've been on the interview side where I've interviewed for a ton of data analyst positions. I was also a hiring manager, and then even before that, I was on a hiring team where we conducted a ton of SQL technical interviews. And so the questions that we're going to look at today are very, very similar to ones that I have seen in the real world or even given myself to interviewees. With that being said, let's sh up on my screen and take a look.

All right, so we're here on analystbuilder.com. We're going to go over here to the questions tab; we're going to filter to the free ones, and then we'll go to the hard ones. Now, there are a lot more hard ones here on Analyst Builder, but under the free tab, uh, we only have three. Looks like I didn't uh get this one right, but we have that one today, well, so we'll see if I get this one right today. You can go try out these questions completely for free, and we're going to be taking a look at temperature fluctuations and Kelly's third purchase. Then there's another one called cake versus pie, which I think may be the hardest of these three, so you might want to go ahead and try to take that one and see if you can get it. Now, this is Kelly's third purchase; we'll start with that one, and then we'll do temperature fluctuations. Uh, we'll see how quickly I can do these two because these are hard, but I think we can do it. So let's look at Kelly's third purchase. It says, at Kelly's ice cream shop, Kelly gives a 33% discount on each customer's third purchase. Write a query to select the third transaction for each customer that received that discount. Output the customer ID, transaction ID, amount, and the amount after the discount, account as discounted amount. Order the output on customer ID in ascending order. Note: transaction IDs occur sequentially; the lowest transaction ID is the earliest ID. Now, that's really important; we'll have to remember that. Now, before we jump in anything, let's um let's go look at the data, but then let's start making some notes. So we have a customer ID, we have the transaction ID, and the amount that they spent. Now, this is the amount that eventually we'll need to use to calculate the end uh amount that they paid with the discount. So they get 33% off whatever this number is on their third purchase, if that if you're tracking that. So what we need to do, and let's start making some notes: one, we're going to need to apply a discount, so that's going to be 33%. We have to identify though the person's third purchase, so when they come in three times, on that third purchase, they then get to get that discount. So how can we do that? Well, I I'm almost certain, just looking at this data, because we have 1 for a customer ID 101, 1, 1, what we can do is we need to order this transaction ID and then give it some type of rank. Now, because each transaction ID should be unique, and we'll double check that, but it should be unique, we should just be able to use row number, but we could also use rank or dense rank, um, but they should give the exact same output for each of them; it shouldn't matter. So I think using just row number, um, and then filtering when it equals three, so we're going to apply a row number based off the customer ID and the transaction ID, and then for each customer we'll give it a row number, and then when it's three, which is the third transaction, that's the one we give the discount to, um, in our output. Let's look at what our output is going to be. Our output is going to be, let's take a look: select the third transaction, output customer ID, transaction ID, amount, so all columns, all columns with uh, and I'll just copy this, discounted amount, so really everything with just that new column, and then we need to order by the customer ID. So we got a lot to do; this this definitely doesn't look like, of course, this is a difficult question, it's a hard one, but this doesn't look like a super straightforward one. So the first thing that we need to do is we have to identify this row number because we cannot apply the discount until we know which data to apply it to. The output in the order by will come at the very end. So let's look at this; let's do um let's do a comma here; we'll come down here; let's do row number. Now, this is a window function; it's, you know, if you haven't used these before, you haven't taken like my full course and and you know, work through these things, row number uh is a window function that's going to apply to a window, or kind of like a group bu is is what I compare it to. When you group bu, all of those customer IDs with 1000 and one are going to be grouped into one row with uh a window function; they aren't going to be grouped into one row; they'll just be in a window where you'll see each row individually, and you can apply something to each row instead of grouping it and aggregating the data. So it's really unique, um, and really useful. So we're going to do row number, and what we need to do, this is over the transaction ID, and we need to order by, order by transaction ID, and that's going to be ascending. So the earliest one, it says lowest transaction ID is the earliest, so we need to start with the earliest, then go to the highest, and pick the third one. So let's just run this, and I need to do this over, I said row number, I didn't write that right at all, so we'll do over, and then we write it. So we're doing, we're applying this row number, the over is the keyword that we use to specify that this is what we are doing it on, and so now we have this, and we can't just do this because it's applying the row number appropriately based off of only the transaction IDs from from lowest to highest. Here's the thing, then we have to do it per each customer, so it's each customer's third, so we have to use partition by before the order by. Now, partition by is going to separate it out by the customer ID; it's kind of that's kind of like the grouping part, um, and so we'll use Partition by customer ID, and why did I copy that by customer ID, and now let's run this, and now when we come down, it should say 1, 1, 101, and notice that we have this um row number applying at the customer ID level, and then when it gets to the last customer ID and it goes to the next one, it restarts. So now this is that third person's transaction 1001, and then at 102, this is the third transaction. Now, here's the tricky part about trying to then use this row number is I cannot come down here and say where, and let's label this, we'll say as rowcor num. Let's run that, uh, whoops, because I have this as blank, let's comment that out real quick. So I have this row num, but I cannot say where row num is equal to three. Let's try it; so it's going to say unknown column; it doesn't understand that that's a column, and you may be thinking, well, you know, in aggregations with Group by, you can use the having statement. Well, let's try the having, and let's run this; it says the window function is allowed only in the select list and order by Clause; we cannot use in the having. So what we need to do is we need to actually make this as uh its own little output is what I'll say. Now, we can do that in two different ways; we can use a CTE and we can use comment table expression to kind of store this data down here how it is, and then we can query off it later, or we can put it in a subquery, or if you know we wanted to get really Advanced and we're using um an actual MySQL database, we could use something like a temporary table or a view or something; we could do other things, but for here, let's wrap all of this in a um let me come right here; let's wrap all this in a subquery, and when you have uh a subquery in a from statement, you have to label it, so you have to give it a name, so we're just going to call as row numbers, and let's select everything, and what this is doing is we're selecting everything from this data right down here, this table that we've essentially created. So what we're going to do is we're going to select everything. Now, what we need to do is we need to say where rowcor num is equal to three, and let's run this. So now we have each person's row num three; this is the third person's transaction. Now, this is really good. So what we need to do next is we need to then calculate this amount, so it gets a 33% discount now. So let's select the columns that we actually want in our output; we need customer_ID, we need transaction ID, we need the amount. Now, we need to calculate the discounted amount. Remember, we have to label this last one um we'll say as discounted amount. So this next column is going to be this calculation. Now, we have to give a 33% discount, so we can't say amount times, let me bring this down like this, we cannot say amount times 0.33. Let's run this, and let me see, I just spelled transaction ID wrong, transaction ID, that's it, always gets me. Um, this is actually a this is 33% of this number; that's what this is. Now, 33% of this number is not a 33% discount; we're giving them a 67% discount. What we want is 67% of the amount. Let's run this; this right here is 33% off the total amount; it's a discount. So instead of paying $94, this person only had to pay 62.98. Now, the last thing we need to do, the very last thing, is order by customer ID, and it looks like it already is, but I'm going to do it anyways; we'll do order by customer ID ascending, and let's run this. This should be our final output, and you know, it took a little bit of work to get there; we had to use this subquery, but I'm pretty sure this is right. Let's go ahead and check this answer, and there we go, our solution is correct. Now, remember, there's other ways to write this; there isn't just one way. Um, this is kind of the difficult part about hard interviews or like senior-level data analyst interviews for for SQL um technical interviews; the difficult thing is there's not only one way to answer it, and so it starts getting down to, okay, what's the best way to solve it? Walk through your thought process. So everything that I just did where I walked through and I said, okay, I could use any of these, but row number makes the most sense for this data; understanding the difference between those and why I'm choosing one over the other, um, is really helpful for the interviewer to understand and gauge kind of your skill level. That's why I recommend you write it out well, but also talk about it; the thought process is kind of the most important part. Now, if you tried this question, you could not get it, or you couldn't solve it, you can always get a hint, um, you can take a look at the expected output, or you can go up to the video explanation where I walk through this entire question, or just go look at the solution, and let's see if I wrote it the same way. Well, I called it RN for row number, but this is essentially the same, although I wrote out the column names; it's essentially the same, but you could have done a CTE with this as well, but that is how you would solve this Kelly's third purchase. I will leave a link in the description if you want to try that one out.

Now, let's go up here; let's go to temperature fluctuations. So this question says: write a query to find all dates with higher temperatures compared to the previous dates (yesterday). Order dates in ascending order. Okay, let's look at the data. So we have our date over here and the temperature. So it looks like, for example, this one, this is the 2nd of January; this temperature was 70; the previous day's was 65. So we want to identify this date, and I think it's just the date, um, to find all the dates with higher temperatures; it looks like our output is just going to be the date, and I'll write that real quick, um, output just date column. Now, how are we going to do this? How are we going to compare this? Uh, initially, there are two things that I think we could do: one, we could use a window function; we could use a lag uh a lag function on this, which would look at the previous rows' data. So if we ordered on the date, which it already looks like it's ordered, we can use the lag function to look at the previous value. So here is 70, then we use a lag function, it would pull over 65. Over here, that perfectly fine uh way to do it. The other way we could do it is we could do a self-join, so we could tie the table to itself, but instead of doing it where the date is equal to the date, we say the date minus one; that's another way that we could solve this. So you can go ahead and try whichever way you would like to. I think I prefer the self-join; um, I just think the last one we did uh a row number, which is a window function, so I don't want to do another window function, right? Although you could solve this with a window function, um, I'm going to try a self-join. So let's pull this over; um, I think I'm going to do a self-join, but on the previous day, where there's a one-day difference, so where the day is one day off is what I'll say, um, then we can use that to compare, so use the temperatures to say where one is higher than the other, and that should make more sense uh in just a second when we start writing it out. But now we also need to order by dates ascending; that's it. So we have our data down here, and in order to do a self-join, we're just going to say um I'm going to say inner join, but we can do if there's a different type of join you want to do, you can also do that, but I'm going to say on temperatures, and we need to label these differently, so we'll do T1 for temperatures one, and then T2. So now it's like we have this table over here, which is temperatures; we have this table over here that's temperatures, and we're going to join them together. Now, what are we going to, oh, let's do T2. Now, what are we joining these together on? Um, we're going to be joining this on the dates. Now, there's a few different ways that we can write this, but there is a function called Date diff where we can take one date and compare it to a different date and make sure it's one day different. So let's go ahead and take a look at that; let's do um a join on, and we'll do Date diff, and then we'll do t1. and it's auto populating it for us, but Date diff, comma T2 Dot, and then we'll say date right there, and it should be a one-day difference. Let's try this; let's run this, and the reason why it's not pulling up is because we have all the same uh column names. Now, when it has the same column name, it's just showing up as just one; one is overlaying the other. So what we're going to do is t1.dat, comma t1.temperature, and let's get rid of that. Then we're going to label these other ones different, so we'll do t2.date as date2, and then we'll copy this and and we'll do t2.temperature as temperature2. Now, let's run this, and let's see what happens. So we have uh this date compared to the previous date; this date compared to the previous date; so 3 versus 2, and let's keep going; 4 to 3, 5 to 4, 6 to 5, and so every single date has the previous date. Now, what we can do is we can compare this temperature to this temperature. Now, we can do that in a where statement, or we could just do it in the join; we can make it a conditional uh part of the condition in the join. Maybe I'll write out both, but let's say and we'll do t1. uh t1.temperature is greater than t2.temperature. So let's run this. So now we have 70 compared to 65, and look like the other one wasn't as high, so the third day is gone; 58 compared to 55; 90 compared to 58; 82 compared to 70; 88 compared to 82. So these are all of our dates right here in this column that this temperature was higher than the previous day's temperature, and so what we should be able to do is just get rid of all these columns and just take the date, and let's run this, and there we go. And all we have to do now is order by, and I think it's already correct, but I'm just going to, I always like to write it out if it's asking us to do it, take do this in ascending, and so let's run this. Yeah, in ascending order, I didn't write that, oh yeah, here it did; I wrote it right here. So now this looks correct to me. Let's go ahead and check our answer, and there we go, our solution is correct. Now, again, there's multiple different ways to solve this. Genuinely, off the top of my head, I could think of two, probably another one, another third option, um, just off the top of my head, cuz I've been using MySQL for a while. Walking through that in your interview, saying, I think I could do it in this way or this way, but here's why I'm choosing this way; that really tells an interviewer this guy knows what he's talking about, or girl, this person knows what they're talking about; they understand it; they get it; I can trust that this person will know how to do the work that we're going to give them if we hire them. You want to give them a lot of confidence; that's all I'm going to say. Now, if if you don't know how to do this, or you've never done something like this before, that's what this platform is for, um, so that when you get into those interviews, or you know, you want to learn and go take a course, when you get into those interviews, you can confidently say, I know this skill, uh, and so if you had trouble with that one, you can always go to hints; you can always go to the expected output, video solution, solution, um, let's see how I solved it here. Oh, I solved it the exact same, but I called it, I could have solved it a different way; I think the lag function would have done just as well; uh, it may been simpler actually. So this is the one that I used, but honestly, the lag function in a window function may even be better. So we solved this Kelly's third purchase; we solved this temperature fluctuations. I will leave links in the description for both of those; go ahead and try those out yourself. And again, even back in the question section, there's this cake versus pie, uh, which is probably the most difficult of the hard ones uh under the free tier. Now, if you go back under the, you know, there's a a where you can pay for a subscription; under those, there's like 20 hard questions, and they're all very unique, very different, focusing on

Data cleaning, window functions, uh, different types of joins—and they're all really unique, uh, and fun to do—but this cake versus pie one is really interesting. I want, um, I want you guys to go try this one. I'll leave this one in the description as well. This really interesting, difficult question. So those were our two hard SQL interview questions; uh, they were pretty challenging. You know, window function and then self-join—two things that are a little bit more complex. And you'll see easy and medium questions, uh, in the next lesson. We'll be solving a very hard question. So if you have not checked out analystbuilder.com, it's one of the best platforms for data analysts. I created all the content on there, all the courses, all the questions, and we have so much more coming to the platform. If you like this video, be sure to like and subscribe below, and I will see you in the next video. [Music]

What's going on, everybody? Welcome back to another video. Today we're going to be solving a very hard SQL technical interview [Music] question. Now, if you've been following along with the entire series, we had videos on easy, medium, and hard SQL technical interview questions, and we solved two questions in each of those videos. But we are on to the very hard questions. These very hard questions are, in fact, very difficult, so I'm only going to be doing one, although there's multiple on the platform that you can try, but I'm just going to be doing one because it's going to take a long time to solve it. Now, practicing these questions is meant to help you learn as well as feel comfortable and get ready for these technical interviews that you're going to get as a data analyst. The easy and medium questions are more geared toward entry-level beginners, or maybe even mid-level for the medium questions. The hard questions are geared more toward mid-level or senior-level data analysts, and then there's the very hard. I don't think that you're going to get a technical interview this difficult. I know I haven't, nor have I ever given one, even though I've given tons of technical interviews before; um, I've never gotten a question this hard. These are more of a challenge, really—a challenge of can you figure this out, uh, because it's pretty difficult. So I hope that you find this really interesting. I want you to try it out. But with that being said, let's jump onto my screen.

All right, so we're here on Analyst Builder. Let's go over to the questions page. Let's filter down to very hard. Now, I will note these are not under the free tier. If you go to the free, the very, very hard ones are not under the free tier. So if you want to try these, these are the very hard ones: Consecutive Visits, Twitter Addiction, Employee Hierarchy, Biggest Spenders, Complex Address, Uber Cancellation Rates. Now, today we're going to be trying this Complex Address, but if you want to try out any of these other ones, head over to analystbuilder.com; they are super, super fun. But let's try out this Complex Address question. It says: You are given a database containing customer addresses. Write a query to break out the address column into separate columns for Street, City, State, and Postal Code. Note: Some addresses may have additional unit or suite information, for example, Suite 5A or Unit B, which should not be included as part of the street. So let's go down here. Let's look at the addresses we have: 1123 Main Street Suite 5A. So that's for example; that Suite 5A should not be included. It says then we have New York—that's the city, uh, I'm guessing that's New York City—then New York, uh, then 12345. Then the same thing: Minneapolis is the city, then we have the state, and we have the ZIP code. I think that's all we need to—yeah, the postal code. So let's start making some notes here. So we have to—the output needs to be, uh, Street, and I should just copy this street all the way down to postal code. There we go. So that's what our output needs to be. There's nothing on ordering; uh, I think the most difficult part is going to be just breaking it out. So breaking everything out. Now, how are we going to do this? There's, um, one main way that I would be doing this, and this is using substring. Now, I've done this a thousand times in my real job. This is an extremely realistic thing; happens all the time. Data comes in just like this, or sometimes separated by commas or just spaces, and you have to figure that out, right? So this is interesting because this is separated by—no, it's just a space. These are separated by these dashes, um, and then we can't include this 5A. So I'm going to use substring, and we'll see if this works, um, but with this we can choose our delimiter, and that's really important. So we can say whether it's a space, whether it's a a dash or something like that, but we also have to note—I'll just write note right here—can't include things like Suite or Unit. So we have to remove that somehow.

Um, let's go ahead and pull up the data over here, and let's take a look. And I wanted to do this in MySQL. Now, you can do this in Python, you can do this in Microsoft SQL. I had started out writing this in PostgreSQL, which, uh, the syntax is actually a little bit—potentially maybe a little bit different—actually, it's different in Microsoft SQL Server, I believe. PostgreSQL, the substring is the same—don't quote me on that—but we're going to be using MySQL because I've been using that throughout the entire series. But if you want to, you can use Python, MySQL, PostgreSQL, Microsoft SQL Server, whichever one you feel comfortable using. Now, let's run this. So I'm going to keep everything here, but I'm going to add to it, so we can confirm that the—it's accurate, like our output is correct. So that's why I'm going to keep the everything there. Now we're going to use this substring, and the first one—one that we have to figure out is, um, this one right here. Now, it should be fairly easy with something like one that does not include Suite 5A, and for example, we would just do substring, and we have to pass through some parameters. The first one that we need to pass through is just the string. Now, this entire string is kept in address. So when I say we're passing through the string, we're passing through a column where it has multiple rows with strings in it; that's all—that's all I'm saying. The next parameter is our delimiter. A delimiter is something that—how are we separating this out from itself? So I'm going to put in quotes, I'm going to put a dash, and then the next parameter is where are we starting? So are we looking at the first delimiter, the second delimiter, the third, fourth, fifth? Because this one has multiple delimiters—this one has one right here, and it has one over here. So if I put a one here, um, we get null, and that's because I wrote substring; we actually need SUBSTRING_INDEX, uh, I'm thinking of Python. In Python, I'd be using substring; in MySQL, I need SUBSTRING_INDEX. There we go. So now we're doing this on the first delimiter right here, but if I change this to two, now we're looking at this delimiter. So that's the first delimiter, second delimiter. So we only want the first delimiter, but here's the issue with this: We have—and there's a unit right here—so we have one that has unit, and we have one that's Suite 5A. Here's what we need to do: We have to get rid of it if it has a Suite 5A or if it has Unit B. So we'll need a case statement for this. Um, what we'll write is we'll write CASE, and this will be our ELSE. So if it has a suite in it or it has a unit in it, we will use the SUBSTRING_INDEX on it, but if it doesn't, we're just going to treat it as normal. This is like our normal one. So I'm going to say—and we'll do it right here—I have to format this in a little bit, but we'll say WHEN—when the address is like—and now we're looking for a pattern; we're going to search—does this pattern exist in this? Now, uh, it's possible in some instances, if you're using like a million rows of data, Suite could be in like 123 Sweet Street. I've never seen that before; um, I don't think we need to take that into account today, but that might be something to consider in a in a real-world example. But this is a real-world example, but we're just going to include Suite, and I'm going to include—and I'm going to use these, um, these wildcards. So this means anything can come before this, anything can come after this; it just has to have a space Suite. Now, you don't have to add the—that you could do it just like that; it makes more sense to me because we actually need to remove that in the future, and I'll explain that in a little bit. But when that happens—when there is a suite in there—what do we want to do? Well, we need to use SUBSTRING_INDEX on it, but we cannot use the dash; the dash has got us in trouble last time; it kept the S 5A in there. What we need to do is we need to use this space as our delimiter. So then when it gets to that delimiter, it's not included, right? So then we come over here, we put the Suite in there. Let's run it, and I got a syntax error, and that's because we need an END, and we need—and we can label it as well, but this will be our street; we have to have an END to signify that the CASE is done. I'm pretty sure that's the issue. There we go, and that fixed it. That's because, like, the delimiter with the dash—when it got to the dash, it took everything before it. So now if we find a suite, we're using that space Suite as the delimiter. Now, if we put this as the delimiter, you may not be able to see it, but there should be a space right here, and that might cause dirty data. I'm going to actually—I'm going to keep it like that; we'll see if my hypothesis, uh, is correct. Now we're going to do the exact same thing, except now we're looking for—which one was it?—Unit. So if includes Suite 5A or Unit B—those are the examples we need to get rid of that—and we'll use Unit. And let's run it, and there we go. So this looks perfect; this looks exactly like what we should be getting in our output. So now, uh, whoops—now we need to come here down to the street. So now we have the street, but we need to get—next we need the city. So the city is right after the street, but here's the thing: It's in the middle, and if you've ever worked with data like this, it's a little bit tricky because, uh, a delimiter only goes to one point, right? So what we can do is use a double, uh, SUBSTRING_INDEX. So we're collecting the SUBSTRING_INDEX, and then, uh, within that text of that SUBSTRING_INDEX, we do another SUBSTRING_INDEX. So I think that's what we need to do here. So let's come down here, and we can just copy this, cuz it's already written out for us, um, and let's run this. Now, what if we say two here? Let's run this; um, we can do it this way and then go backward. We can do a minus. So in SUBSTRING_INDEX, instead of a positive one looking forward—that's starting from, um, the left-hand side of the string and looking this way for the first one—we can do negative, which starts from the right-hand side and looks this way. So left to right when it's positive, right to left when it's negative. So then we can wrap this as a SUBSTRING_INDEX, and now this whole thing right here is our string, which is this—this is our string. So now we're going to look backward; we're going to do minus one, um, actually our our delimiter first is a dash because we're looking to this dash, and then we need to go to one, and let's try this out, and there we go: New York, Minneapolis, Goldsboro, Maples, Flowertown. So this looks perfect. So let's keep that exactly how we have it. Now, for the next one that we need—and I'm going to label this as City—now we're going to copy this whole thing, bring it right down here; we'll do this as State. Now, obviously this isn't going to be our answer, but we can run it. So here's what we need to do: Before we took this whole string, and we got here, and then we went backward to this delimiter and selected New York; what we now need to do is we need to go—we need to select this New York, Minneapolis. Now, there's a space here, and so what we should do is we should go backward to this index right there. I think we need to do that first. So we'll use that as our our starting place, and so we'll do negative one, um, yeah, so I'll just do it. So we'll go negative one, but then we need to go forward. So this is going to be our string right here: New York 125. So then we need to go forward, and our delimiter should be a space. Let's do a space right here. Let's run this, and there we go: We have New York, Minneapolis, North Carolina, Massachusetts, I think, and Florida. So that's our state, and we have one more, and that's going to be our very last one. Now, this one should actually be a little bit simpler because we're just starting at the very end; it's not in the middle, which is a little bit tricky, right? So we're going to come in here; we're going to say AS, and this is Postal Code. I really want to stick to exactly what they told us to call them; I don't want to go changing it. So now we're looking for a space delimiter, but we're looking backward—negative one. So we're starting from the right-hand side and going to the first space, and that should give us that 12345. Let's run this, and there we go. Now I'm going to get rid of everything, and let me actually come up here; I'm going to get rid of everything, and let's run this. So now we have the Street, City, State, and Postal Code. This all looks correct. I don't know—I genuinely don't know if I check this answer if it's going to be correct or not. My hypothesis—what I think is going to happen—is—is—is not going to be correct because I think somewhere in our output we have some spaces that we can't see. For example, this Suite—if we're using just the Suite and not the space Suite—I think we have an extra space at the end of this one right here. So it goes Suite space, which I don't think is correct, and it shouldn't—you shouldn't have that in actual output. In a real database, you don't want, uh, uh, leading or trailing spaces; that's dirty data. So let's try checking it, and there we go—our answer is wrong. Let's try just fixing this and see if that's it. If not, it could also be one of these. Well, let's try this one. Now, ah, there we go. So that is—that was the exact issue, and it's really hard to catch if you're not looking for it; you may not catch it because it is a little bit tricky, um, but that is—I—let me see if I added this in as a hint; maybe I did, maybe I didn't—but that is something that you need to account for in real data. So, um, I know that there were people in the Discord—if you haven't joined Analyst Builder and tried out these questions—um, we have a—we have a Discord with like 2,000 people in it, and this question—when people have been trying to solve it—have been really having issues with this exact thing—is like, I—it looks correct, but it's not correct, um, that is a real-world solution, a real-world issue, and so that is how you solve this question. Now, you know, I have a whole video explanation on how to do this, um, as well as you can just come down here; let's look at the solution. So yeah, this is exactly how I wrote it; um, there are other ways to write this, actually, but that is how you can solve it. Now remember, you can come in here, even in the—in the questions—and go to the difficulty, and you can take a look; you can check out all these other questions, um, and see what they look like and what the questions are. These are really tough. U—for example, this Uber Cancellation Rates is a really interesting one: Find the cancellation rates of requests with unbanned users; both client and driver must not be banned each day between 2023-12-23 and 12-25. Round the cancellation rates to two decimal points. So this is another really interesting question, um, and there's actually two tables here. I'm not doing this one today, uh, but these are more questions. I'll do more of these videos in the future, just because I really love them; they're super fun, um, but if you want to try that question out, I will leave a link in the description; you can go ahead and check that out. If you have not tried out Analyst Builder already, I highly recommend it. I created all the content myself; all the questions, all the courses are all done by me. So if you like my YouTube channel, you will love Analyst Builder's all premium, really high, uh, quality content. So go ahead and try that out. Thank you guys so much for watching; I really appreciate it. If you like this video, be sure to like and subscribe below, and I'll see you in the next video. [Music]

What's going on, everybody? Welcome back to another video. Today we're going to be starting our Azure [Music] series. Now, if you don't know, Azure is a cloud-based platform owned by Microsoft. Azure is one of the biggest cloud platforms and has millions of users all around the world. I myself used it for many years, and I absolutely love Azure. I think Azure is a fantastic cloud platform, and that's why we're going to be starting to learn it. I think knowing a cloud-based platform is an essential skill today for any data analyst, data scientist, data engineer out there. So that's what this entire series is going to be for: We're going to get our account set up; we'll look at things like account storage, SQL databases, uh, even some things like data pipelines—so using Azure Data Factory and Azure Synapse Analytics. These resources and tools within Azure are ones you're absolutely going to see, and so these are the ones that we're going to be focusing on. In this lesson, we're just going to be getting everything set up. So we're going to be creating a Microsoft account; we're going to be creating an Azure account, and then we'll be doing a walkthrough of the UI, just so you can kind of see it and get familiar with it. So with all that being said, let's jump on my screen and take a look.

We're going to start right here on the Azure microsoft.com/free over here. Now, this is going to be a little bit different if you're in a different country, but this is for, uh, me here in the US, but I'll have this link in the description so that you can come to it; you can create your account, and then we'll get into all the, uh, UI and how everything looks once we actually set up our account. Now, before we actually create it, just go down here: Here are some of the things that you're going to get when you create your account: You're going to get, uh, popular services free for 12 months, which is really, really great; 25 services that you're always going to be free; and then we're going to get this credit right here. Now, this credit is really important because some of the things that we're going to be looking at in this series are not things that are completely free all the time. So, uh, we're going to be using some of this credit throughout this series, so you want to make sure you can get that; you just need to create a new account, and that's what we're going to be doing, and then we don't have to use our money; we can use Azure's money, and we will all thank them for that. Let's go over here; we're going to click on Start free. So we're going to start our free trial. Now, if you already have an account, you can use it; if not, we're going to click on Use another account. Now, again, for this, you can sign in with things like GitHub; you can sign in with a previous account.

If you do not have an account, I'm just going to show you how to create one really quickly, and that's what we'll use going forward. If you already have an account, if you're already signed in, go ahead and skip a little bit forward, maybe like a minute or so, and we should be done creating this account. But let's go ahead and see how we can do one. We're going to go to create one; we're going to call, call this uh, Alex the Analyst at outlook.com, and there we go. So we have Outlook, or we can do hotmail. Hotmail feels like it's from like the '90s, so I'm going to do Outlook; at least it feels uh, somewhat current. Then we need to create our password, and I don't want any tips, so I'm going to do a password right here, and let's go ahead and click next. You're going to fill in when you were born, and we'll go ahead and click next, and it should be creating our account. Looks like we need to do a little uh, puzzle here; looks like we need to rotate it where it's pointing, and there we go. So now we're signed into our Microsoft account, but now we need to actually create an Azure account, and those are two separate things, so you need two accounts. So we're going to go through; we're going to fill in all this information: your name, your phone number, uh, your address, and all these different things. Then we're going to have to fill out uh, our card, so a debit card or credit card, so that we can actually, if we go over the $200 or if we use a service that is not free, then it is going to charge us. We won't be doing that, thank goodness, but you do need to have a card on file in case you do that. Let's go ahead and fill out all this information, then we'll go to the next part.

All right, so we just created our profile. Now we need to identify our verification by card. One thing I will note: in the profile, I did have to verify my phone number, so they sent me a text; I put in the number, and there you go. Next thing we need to do is we need to put in our debit card, so go ahead and do that. And once we fill that out, you're going to see this right here; this is a little welcome to Microsoft. You can go ahead and look through this if you'd like; I'm just going to get out of it because we don't really need it. Now this is going to be the first thing that you see; this is just the homepage of Azure. This is where you can access different services and resources and look at your profile and notifications and all these different things. Now the first thing that we're going to do is just take a look at some of these resources, because there are a lot of resources within Azure. And so if we come down right here, let's go a and click into one of these; this is Azure Synapse Analytics. If we want to uh, actually use Azure Synapse Analytics, we need to create a Synapse workspace, then we can start using this resource and start using this application, and all we would have to do is create this, so we'd come in here to create Synapse workspace, we would start filling out all these things, and then we would actually have access to that resource. And so this one specifically is one that we will be looking at in this series. Uh, let's go down and just really quickly take a look at some of these; we're going to be looking at Data Factory, so creating workflows and data pipelines. We will be using Azure Synapse Analytics, and if we go down to databases, we'll take a look at SQL databases as well as maybe one or two others, because knowing how to use databases uh, with the resources within Azure is actually quite important. And then if we go down to storage, way down here, let's keep going; if we go into storage, we will be looking at storage accounts, and that's actually where you can access something like a data lake. And so this can be pretty intimidating just looking at this because there's so many different things, but in this series we'll really focus in on the things that I think you need to know that you're definitely going to be using as uh, data professional, or especially a data analyst, but you know, even things like uh, let's come back up here, even things like data factories are used by data engineers, database developers, data scientists; it's everybody. And so knowing how to use these tools uh, with an Azure is extremely important, so that's what we're going to be focusing on in the next several lessons. We'll be getting into specific resources within Azure and how to use them. That is all we're going to take a look at in this lesson, because we're just getting set up, creating the accounts, looking at some of kind of the user interface of how Azure actually looks, and in the next several lessons we'll be diving in, getting hands-on experience with a lot of these tools. Thank you guys so much for watching; I really appreciate it. If you have not checked it out already, I have a full course on AWS and Azure on analystbuilder.com; be sure to go check it out. I'll leave a link in the description. If you like this video, be sure to like and subscribe, and I will see you in the next video. [Music]

What's going on everybody? Welcome back to another video. Today we're going to be taking a look at account storage in [Music] Azure. Now account storage is super important with an Azure; this is where a lot of companies are going to store a lot of their data. Using account storage in Azure is a super flexible way to store your data; it can store just about anything, and you can even upgrade it into a data lake. And so we'll be taking a look at all these things in this lesson, just to get you familiar with how to use account storage, because you absolutely will be using that within Azure. Without further ado, let's jump on my screen and take a look.

So let's start by taking a look at our resources right here in storage; we have our storage accounts. Now as you can see, there's lots of different options for storage, but by far the one that you're probably going to use the most is storage accounts. Now we do have right over here; you'll notice Data Lake Storage Gen1. If we click into this, and let's say we wanted to create a data lake, it says right up here: Azure Data Lake Storage Gen1 will be retired on February 29th of 2024. We recommend that you migrate your Azure Data Lake Storage Gen1 to Azure Data Lake Storage Gen2, and that is located in the storage account. So let's come back here; we're going to go right into storage accounts, and let's really quickly set up a storage account. Now this is some helpful information if you've never used a storage account before; it says you can store about 500 terabytes; it has general purpose storage, use for object stores, NoSQL data store; uh, you can define and use queries for message processing, and you can also set up file shares. At the very end, we have blob storage accounts for hot and cool access tiers; these are all things that we'll look at uh, in this lesson, and so that is just kind of a preview of a lot of the things that we're going to be looking at. So let's go ahead and create our storage account. Now this is a brand new account uh, as we set it up in the last lesson, so I'm going to be walking through this with you as if I have a brand new account. Now you have to have your subscription, which if you're using the free tier, you'll have a free Azure subscription, but we have to have a resource group. Now we haven't created a resource group; let's go ahead and create a new one, and we'll just call this one Alex the analyst, and we'll click okay. Next, we need to specify storage account name; we'll call this one Alex the analyst storage. I'm guessing that's unique; the storage account name has to be unique across all of Azure, so if you type in something kind of generic, usually it will already have been taken, so you have to do something pretty specific. Now this part and the next part uh, these are pretty important. This region says: choose the a region that's right for you and your customers; not all storage account configurations are available in all regions. Now if you're just setting this up for yourself, just to store some data on the cloud or have a little app that you're creating or something like that, it's very easy; you're just going to select your local region, and for me that's US East. But what if you have a company or a product or an app that's being used by people all around the world? For example, you have a client-facing app that they're using, and they're going in and they're uh, retrieving data from some type of storage account, or maybe they're running a query on your website of data that's stored in a storage account, and what's happening is is it's stored locally to you, but they're way over here in uh, let's say they're in central India. And so in order to get from central India to the US, it's going to take a lot longer to retrieve that data, and so it could take 5, 10, 15, 20 seconds for it to get to them in India versus if it's just locally in US East. And so you need to know where your customers or where your clients are located who are going to be using this. Now just for this lesson, we're going to keep it in East US because that's where I am located, um, but that is something you really need to consider, especially as you get more advanced with using Azure. But you know, just for the basics, you know, you don't really need to be thinking about that; I just want to walk you through my thought process as we're going through this. Next, we have to specify our performance; this is just going to determine how quickly and how easily you can retrieve your data. We're just going to go with standard, but premium would just allow you to retrieve that data even quicker. Next, we have redundancy. Now redundancy is important, and it's one of the benefits of the cloud, which is if a server goes out that's storing your data, they're going to have a backup of that data somewhere else, and you can specify where that is going to be located. You can either do it uh, a locally redundant storage, a geo-redundant storage, a zone-redundant storage, or a geo-zone redundant storage. And so if your data is super important, it is critical to what your website does, it is critical to your clients; if it was deleted by accident in any way or destroyed in anyway, your whole company would collapse; you're going to choose this one down here. But if it's just some local files that you're uploading for someone to pick up uh, and it's not that important, you're going to do something like locally redundant storage. And so that, that's what we'll choose uh, just something to consider though. Now there are other things you can do in advanced networking, data protection, encryption, tags, and all these different things, but if I'm being honest, 99% of the time you're never going to use any of these unless you really, really, really know what you're doing or that's your job; you're some type of database administrator creating these storage accounts for people on your team; most likely you're never doing that. So let's go down here; we're going to click create, and it's going to say we have our deployment in progress, so it's going to start creating the resources needed for that storage account, and then we're going to actually get into the storage account and start using it.

And just like that, took about 10 seconds; it says that your deployment is complete. We don't need to go to the resource up here; we're going to go to the resource right here. Now what we're looking at is the user interface for this specific storage account, so we have Alex the analyst storage. If we go back, back to all services, we come into storage accounts; you can now see that we have one. We don't have to just have that uh, boilerplate text and then create; if we want to create another one, we'll come right up here, but we can come into this storage account, and we can look at this overview. So this is just some of the information on the location, the subscription, uh, the subscription ID, the type of performance, the replication or redundancy, and a few other things as well. Now there's a ton of things on this sidebar, on this left-hand side; we have things like activity log, tags, diagnosis, solve problems, access, data migration, events, storage browser, storage move, and all these different options; it's kind of overwhelming, but I know just from experience that you're not going to use almost any of these. In fact, things like monitoring are typically used like something like logs; they typically used by IT; if there's an issue, most likely you're not going to be coming in here and taking a look at all these things or creating uh, different alerts. You may be working with metrics if you're in the IT department, but again, a lot of this stuff you're not going to be working with specifically most of the time. We're going to be here in this storage browser, and this is where the data is actually uploaded, stored, and accessed. So if you come in here, let's come up here to a blob container; let's say we want to create a blob container just to store a bunch of data; maybe it's for a client or an application or whatever; we just want to be able to store that data in the cloud, and this is kind of the simplest version of being able to use the storage browser or just, just storage in general. So what we can do is we can add a container, and we're going to name this container; we'll call this uh, ATA container; there we go. And if we come down here, we do have an advanced tab; you're most likely not going to use it because this is uh, deals with encryption and just, and this is completely, you know, as a sidebar above and beyond what you probably need to know. But when you start trying to access data later on in different applications, whether it's a SQL database or you're using it in Power BI or whatever you're using uh, this data for, if you haven't encrypted, you're going to have to have a way to unencrypt it within Azure, and so it is an additional level of security, but it makes it a little bit more difficult to access that data later on if it's not super sensitive data. So just something to think about. Let's go ahead and create this, and now we have this ATA container, so we're doing great over here. So we've already created a storage account; we've gone into the storage browser; we've created a container, and these blob containers are amazing; I've used them thousands of times for so many different things, and blob containers are really just used for anything; anything you need, you can use, and you can dump inside of a blob container, whether it's structured, semi-structured, or unstructured data. And so let's see how that actually works. So let's go into this, and we have nothing in here, and we want to upload some data. So let's come up here to upload, and we're going to go and browse for some files. So in here we have a bunch of different files; we have a SQL text file, a PNG, which is just image, uh, a Jupyter notebook file, and a CSV; all these are completely different; none of these are similar almost any way. And so what we're going to do is we're going to select all these; we're going to open these up, and we can upload these. But really quickly, let's come in here and take a look at some of these advanced options, because this part actually is something that you might use. Now really the most important one in here is this access tier; if we hover over it, it's this piece that's kind of important; it says: optimize storage costs by placing your data in the appropriate access tier, and you can come over here and look at all the access tiers if you'd like, but let's take a look at what these access tiers look like and what they actually do. We have four options: we have hot, cool, cold, and archive. Now this refers to how the data is actually going to be stored in the blob storage; if it is hot, that means that you can just retrieve it anytime you want right away within milliseconds; it's just going to be ready to go, and it's going to be there for you. But if you go with an option like cool or cold, it's not going to be there hot and ready just to be able to pick up and use that data; it's going to be sitting in a data store where if you want to retrieve it, you may have to wait a little bit. And so these options are actually a lot more cost-effective, because if you're not using that data actively, you can just plop it in there as a data store where you're not using it for any application or any project, and you have the data stored securely, but you don't have to pay a ton of money for it, whereas if you store it in hot, it's going to cost more money to store that. Lastly is archive, and archive means you most likely won't ever use it; this is a contract that was signed uh, 6 years ago; you need it on file, but most likely you're never going to use this; if you do, you're willing to wait 5 or 10 minutes because it's not going to be an emergency to get that data or that file or whatever it is. And so these are the different tiers that you can use. Now we'll just use hot because that is the default option, but if you have a use case where it's not important that that data is quickly accessible, you don't need it right away, then these other options are going to be a lot cheaper. So let's come down here; we're going to go ahead and upload these files, and there we go. Now what we're going to do is we're going to actually upload one more, and this is going to be for a future lesson when we actually access some of this data; we're going to browse for our files, and we're going to select CSV file 2; I just made a copy of this, and all we're going to do is we're going to open this up, and for this we're going to choose uh, you can do any of these honestly, but let's put it in archive just to be have the most dramatic effect. Let's go ahead and upload this, and as you can see here in this access tier, we have hot, inferred, inferred, inferred, and then we have archive. And so later on in a future lesson when we try to access some of this data, we're going to try to access both of these, and you'll see what actually happens when you have data stored in archive or cool or cold; it's quite similar, and we'll see how these are retrieved. Now the next thing I want to take a look at is right over here under settings. Now under settings, you'll see this Data Lake Gen2 upgrade; let's go ahead and click on this; it says that you can upgrade to a storage account with Azure Data Lake Gen2 capabilities. So if you need things like data analytics and big data storage, you should consider upgrading to Azure Data Lake Gen2. Now we're not diving into data lakes within Azure; I may do that in a future lesson, but this is where you can access to create a data lake; you can upgrade your storage account into a data lake. They used to have a completely separate Data Lake Gen1, which is what we just looked at earlier, but now, now this is all located within the storage account, so you just upgrade from this location, then you'll have those Data Lake capabilities. So that's just something that I wanted to mention while we're here. The last thing that I actually want to look at within the storage account is actually the IAM, which is the access control. Now within here, you're going to have complete access to this because if you come over here and view your access, you can see that you're going to have grants full access to manage everything, and if you come over here, you can read it more; you, you have access to everything because you created it. But what if we want other people to have access to this? Because right now this is a private account; nobody else can access this. If you want other people to have access to this, you're just going to go to add; you can add a role assignment or a co-administrator. Let's just say we're going to add a role assignment; you can

Say this person has the ability to read, but you can't make any changes. So I'm going to click on this one; then I'm going to come up here to members. Now, right now, I am the only person in my Azure account, so so if I wanted to add somebody, let's say Bob, if I wanted to add Bob, he'd have to have an Azure account, but I would click on Bob and I'd say, okay, I'm going to give Bob this access. I'm going to select him, and we're going to give him just the read access. We don't want him to, you know, delete all of our files by accident; he's not the brightest uh bulb of the bunch, so we're just going to give him that access. Review and assign, and we'll go ahead and add that role assignment. Now you can see that my access—I am an owner, but I'm also a reader. And so that's how you grant access to storage accounts. You can also create roles, deny assignments, uh, create classic administrators, but typically this is done by a database administrator, but it is something that is extremely, extremely frustrating about Azure just in general, because any single tiny thing you want to do with an Azure, you're going to have to request access. And so when you're first getting started up at a company, they're going to give you a lot of the base access; they're going to, you know, create your account; they're going to give you some access to Power BI or the data lake or a SQL database. But whenever you want to use anything outside of that, you have to request IAM access. And so that's just something I want you to be aware of, because if you want access to specific storage accounts, you're saying, "Oh, this team, you know, wants me to work on their data or use something with their data, but I don't have access to it," that's because you weren't given access to it; you just have to request it or go to your database administrator, whoever runs that, to ask for permission.

And that's kind of the nuts and bolts of what most people are going to use storage accounts for. There are things like file shares, queues, tables—honestly, I, you don't use them that much, and so I'm not going to dive into it. These blob containers within storage accounts, the data lake, which uh is an upgradeable option—these are kind of the more important things. And so knowing how to store, where to store, and all these uh different options is very important, and so we'll be coming back to some of this data or putting in new data for different lessons when we actually start accessing data within a storage account. So I hope that that was helpful. If you have not already, have a full course on Azure and AWS over on analybuilder.com. I will leave a link in the description if you want to check it out. If you like this video, be sure to like and subscribe. I will see you in the next [Music] video.

What's going on, everybody? Welcome back to another video. Today we're going to be taking a look at SQL databases in Azure [Music]. By now, I think you all know how much I love SQL. I think it is one of the best skills for any data professional to have, but using it in the cloud is a little bit different than using it on your local computer. So in this lesson, we're going to see how you can use a SQL database in Azure. Without further ado, let's jump on my screen and take a look.

All right, so the first thing that we're going to do is we're going to come right in here into databases under the resources. Now we have a lot of different options in here, and there's a ton of different databases that you can choose from, and it kind of depends on what your company does. I'm only going to be showing you the SQL databases, but other popular ones are things like using MySQL or PostgreSQL with flexible servers, as well as things like Azure Cosmos DB. They all have different use cases, and they all have different ways uh that they are implemented, but by far the most common, the one that I've used the most in my career, is SQL databases. So let's come right in here, and what we need to do is we need to create a SQL database. And let's click on create SQL database, and let's actually create it, and then we'll see how we can use it. So what we're going to do is come right down here to subscription. We have to select our Resource Group, which you should have already created. Let's create a database name; let's call this Alex the Analyst DB for database. Now we have to select a server, but we haven't created a server, so we need to create a new server, and again I'm going to call this uh we'll do ATA for Alex the Analyst. I'll call this server—it looks like this needs to be lowercase—let's do ata-server, and then we'll do uh YT at the end. Have to make it unique. There we go. All right, we're overcoming some hurdles here. Next, we have to choose an authentication method. We can use the Microsoft Entra only authentication, SQL and the Entra authentication, or just SQL authentication. Now what that means is is if you come in here and set your uh Microsoft Entra admin, you can set it as yourself, and that does help if you're already signed into Azure; you're going to be using it within Azure. This can be very helpful; it's actually kind of the default method. Let's get out of here. This is the default method. If you come down here though, you can also create an admin login and a password. Both of these have their, you know, place, and sometimes you need to use both, um, and so just choose the one, the authentication method that you want for that server. For us, I think we're just going to stick with the uh Entra admin; we can always change that if we want to. So let's go ahead, and we're going to select ourselves here, going to select that, and there we go. Let's go and click okay, and that should create our server, and there it goes. And now we need to finish creating our SQL database. So you want to use a SQL elastic pool? If you look at this tool tip right here, basically helps you manage your resources, but we're not going to be looking at the elastic pools for our workload environment; we're just going to choose development. Production is going to be a lot faster, because if it's in a production environment, you're going to want better speed, better compute, all these different things. Development is going to be a little bit slower. We can also choose a cheaper database or a more budget-friendly database, so we don't have to uh choose what it gives us; we can come in here and we can define this. So maybe you want to have a provisioned tier instead of serverless, which I don't really recommend; serverless is uh quite nice for scalability, but we're just going to keep it at is. But if you want to come in here and change some of this configuration, you're free to do that; uh, just don't, you know, if you don't know what it is, I wouldn't mess with it. Let's come back to the create SQL databases, and then we have our backup storage redundancy. You can either do locally redundant storage or geo-redundant storage; we're just going to stick with our local. Let's go ahead to review and create. It's going to tell us our cost, which is very, very, very low. Um, if you don't even have the free $200 that they're giving you, which will be uh which can be used for this, it's going to cost you like a dollar uh for, you know, what we're going to be doing, or maybe even like 10 cents; I'm being honest. Let's go ahead and create this. It's going to take a little bit of time to set up all these servers and all the databases and all those things, and then once it is done, we'll take a look.

All right, so our deployment is complete. You can come in here and look at some of the details. We created SQL databases, the server, SQL Server, SQL server, and so all these things are ready to go. Let's go ahead and click on go to Resource and let's exit out of this. All this information is just an overview of our SQL database. Now we have down here some of the more important things that we're going to be taking a look at. We're not going to be diving into all of them because uh I'm just going to show you the most common way. Now we have configure access, connect to application, and start developing. Now in the real world, when people are actually using these SQL databases and when they are getting in here and setting everything up, you can do this in a few different ways. One is you can connect to a MySQL database, and this is a very common practice where they connect it to a database management system; it could be MySQL Workbench or a ton of others that are out there, and you can do that by configuring it, and you're going to get some of that information; you're going to plug it in and connect it. What we're going to be looking at is not that option, although uh that is something that happens often. I'm going to show you Azure's tool for this, and it's going to be open Azure Data Studio. So we're going to open up Azure Data Studio; we're going to click on this right here, and you're going to need to download the Azure Data Studio. Now I already have this, so I'm going to come down here and go to Azure Data Studio, and this should resemble a few different things; things it should resemble a little bit of Visual Studio Code, and it should resemble something like Microsoft SQL Server. It's kind of a combination of both. You have a search, you have some notebooks that you can use, different projects, an Explorer, Source control, extensions—it has a ton of stuff. And so this was really popular when I was using Azure; everybody used this, um, as well as sometimes we connected to MySQL databases or uh Microsoft SQL Server databases and just use those database management systems, but often times we would have everything in Azure Data Studio. So let's come right up here; we are going to connect. Now we have to specify our server name. Let's go back really quickly; we're going to come right over here; we're going to go back to uh this right here; we need to select our server. So this is our server; it's ata-server-YT. We could even come in here into the server and we can just uh copy this if we want to, but we're going to get that. We have our server; we have our Windows authentication type, and so it can either be a SQL login, a Windows authentication, but we chose the Microsoft Entra ID. Now right here is recognizing the analystbuilder@outlook.com; that was for the course that I have on analystbuilder for AWS and Azure. We need to add in our alex.the.analyst@outlook.com. So let's come in here, and we need to sign into our account. So let's go ahead and sign in, and there you go; your account was added successfully. Let's go back, and there we go. Now we're signed in, and we need to select our database. Now it's not popping up the database right away, which should be called like Alex the Analyst DB or something like that. Let's go ahead and try to connect and see what happens. It looks like we're getting an error here. I think let's actually come back here; I think our server name is actually this one right here. I just chose the actual server, but this is the connection that we actually need to make. So uh I'm actually quite certain about that. Let's go ahead and click on this, and now it's saying our connection was denied since "Deny public network access" is set to yes. So this is something I was waiting to see because because we need to configure this just a little bit. So let's come over here, and we need to go to configure access. So we're going to select configure. It says "Public network access" is disabled, but we can enable this, and then what we can do is we can add in our IP address. So we're going to add in your client IP V4 address, and all we have to do is click save. So now it's going to update, and it's going to say, okay, you can access this. Let's not, you know, get too crazy and too wild here. Now we can go back, and we're going to select this, and we already have that selected. So now you can see that we have our two databases: we have the master, which is the one that you're going to get, and then we have the one that we actually created. So all of that to show that you do have to configure a few things, make sure you're doing it properly, and now we can come down here and we can connect to it. And so now right in here we come into our tables; we don't have any tables, but we can come into these tables and views and all of these different things. Now we can actually use this. So now we are connected to our resource; we're connected to our server, and we can actually access the databases, create them, do all of our querying, all of our uh things that we need to do with our data, and we have all these options on the left-hand side. Now this isn't an Azure Data Studio tutorial, uh, but there's tons of stuff that you can do in here. So if you've used something like Azure Data Studio or if you've used Microsoft SQL Server, this should seem really familiar; you should feel right at home. Now I want you to be able to actually use this; this I don't just want this to be something pretty, so I need to show you one other thing that you need to do. Let's go ahead and try to create a table here. Right down here we have this script: create new table. We can keep it new table just with the one column; uh, it doesn't really matter. Let's just say this is ready to go. Let's go ahead and publish these changes. Then we're going to come down here, and we're going to update our database. Now we can come over here, and we have a new table. So we can actually uh open this up; we'll select the top 1000, and of course we don't have any data in it, but uh we have a working table. So now this table is being stored on a server in a cloud, and this is great. So if you've ever used something like Microsoft SQL Server, you have all these tables and databases and all these things you're working with; that's how it's actually used in the real world, except you'd probably see a ton more tables; you'd have access to a bunch of different servers for different clients and different uh data that worked perfect. And what you can now do is let's do ctrl+N, just get a new query window available. I'm going to paste in here just like this. Um, we're already selecting our database; we don't have to say "use this database," go—I'm just, you know, that's what I'm used to, so I'm going to keep it in there uh for any, you know, if you have a put this in a stored procedure or something like that, I don't want to fail out, but this is just a super simple table, um, and we're going to go ahead and run this. Looks like that should be done. Let's refresh, or actually refresh this table, and we have this product. So let's open this up, and now we have data in here. This is our little underlined in red; sometimes if you do Ctrl+Alt+R, it'll refresh it, or maybe it's Ctrl+Shift+R. That's okay; it'll get rid of it eventually; it just doesn't recognize it uh yet, but it will. Um, so anyways, we have our data in here, and now we can write regular queries. And so uh this isn't a SQL lesson; I'm not going to show you how to write SQL, but I have hundreds of other lessons and courses on how to learn SQL, and so this is uh kind of the nuts and bolts of how you set everything up, and this is how people actually use it. So this shouldn't be too intimidating if you know how to use MySQL Workbench or Microsoft SQL Server. And so uh that is really awesome. Now if we come back here, we're just going to take a look at a few more things. This is just within our server, but we don't want to look um at our server; we want to go back to our database. We were to look at a little bit of configuration and Azure Data Studio. Again, remember if you need to uh connect it to MySQL or something like that. Now just within the SQL database, there are a few other things that you can do. One, they have something called a query editor, um, where you can come in here and you can query off of, let's say you have tables; you can query off of in here. I can assure you that almost nobody ever uses this, almost ever; this is not really something that people use; it's there to kind of test connections sometimes. So if you're just setting up a new server or a new—so if you're just setting up a new server or new database or whatever it is, you can kind of check to make sure it's working, but you won't use this in your real work; it's just not, doesn't make sense. Now let's come over here and take a look at this left-hand side; there are some interesting things that I want to show you. One is this Power Platform. When you start working with a large amount of data, you have a server and a database set up, and you're using it, and you have tons of real data in there; you're like, okay, now it's time to connect this, well you can use things like Power BI, Power Apps, and Power Automate; all these things just kind of automatically integrate into it. There's also integrations, but just knowing how you—these are actually used, you most likely won't use them, but just knowing kind of what these are and how they work, you most likely won't use these too much for people like you and I. Power BI is something that we'll probably use quite a bit, and we most likely won't use monitoring too much, but I will say I've had to come into the monitoring quite a bit over my years to debug a bunch of stuff. So if a stored procedure is failing, if the database is failing, and you know you need to figure it out, you can come into the logs. If you need to see how much compute, how many resources you're using, you can look at the metrics. So there are some reasons to come in here; typically though, this is more IT related; this isn't as much of what a data analyst will typically do, unless you work like I said in IT, um, where they monitor a lot of those things to keep cost down and keep things running and, you know, going smoothly. So I hope that that was helpful. I really appreciate you guys watching. If you have not already, be sure to check out my full AWS and Azure course on analybuilder.com, and if you like this video, be sure to like and subscribe below. I will see you in the next video [Music].

What's going on, everybody? Welcome back to another video. Today we're going to be taking a look at Azure Data Factory in [Music] Azure. Azure Data Factory is a super cool tool within Azure because it allows you to create data pipelines and different workflows to extract data and clean data, transform, and put it places. And so it does a lot of different things; it's one of those skills that I started using when I started getting a little bit more advanced in Azure, but I don't think you have to wait till you're really advanced as an analyst or a data scientist or engineer; I think you should start learning it now because it's a really, really great skill to know. So with all that being said, let's jump on my screen and take a look.

All right, so let's get started by coming right down here to analytics, and under here we have data factories. Let's go ahead and click on it; let's create a data factory. So let's come in here; we need to give it a name, choose our Resource Group, and we'll be rocket and rolling. So let's call this the Alex the Analyst, and we'll do ADF just like that. For the region, we're good, and for version, we only have one option, so let's go and review and create this. Now, as we know, there's going to be uh deploying this; it's going to take just a minute, then it'll be done, and then we'll get going. And there we go; that literally took maybe 15 seconds, and so uh just set up the data factory version.

Two, and let's go to our resource source. And there we go. Now this should look, uh, you know, fairly straightforward. We just have some of this information up here. We have our launch studio, and then down here we have some of the monitoring. So when you're actually running these automated systems pipelines, everything that we're going to be doing, um, you have some data on it, and you can see some of the, uh, data on that. Now they do have some quick starts, tutorials, template galleries, and training modules. Go ahead and take those because I've looked at a lot of these, and they're really great.

Um, what we're going to be doing in, uh, this lesson, which we're going to cover a lot of stuff, is I'm going to help get you set up. I'm going to help show you how to do different things, and there's going to be a lot of stuff that we cover, so I'm going to be moving pretty quick, but let's go ahead and launch our studio, and here we go. So this is the Azure Data Factory. There's a bunch of different things that we can do in here. We can ingest data; we can orchestrate, so create code-free data pipelines; and we can transform data. Now we're going to be looking at a lot of this, but not all of it, so stick with me.

The first thing that we are going to do is we're going to work on an ingestion because being able to pull data in is actually a pretty important thing to know how to do. We're just going to select "Run once," and we're going to select "Next." Now, uh, there's a lot of different places that you can ingest data from where your source data is stored, but for us, we are going to select the data that we put in the last lesson. So in our last lesson, we have put some data, um, in a SQL database, which this is our server right here: ATA server YT, and we have this table right here. So we're going to ingest this data. And so what we're going to do is we're going to come over here; we're going to type in, uh, SQL. Should have Azure SQL Database; should be right there. And we don't have a connection yet, so let's just select "New connection." This is basically like connecting to the Azure Data Studio, uh, before, and so this should seem really, really familiar. So we have our ATA server YT; we have our Alex the analyst DB. And so now we should, uh, for our Azure subscription, we have our subscription, and let me go back. CU. Now it's not remembering, uh, so what we need to do now is we need to select our, uh, system-assigned managed identity. So our managed identity is this one right here: Alex the analyst ADF. Now that's going to be really important in just a second because this is by far the most confusing and, you know, frustrating part of doing this if you've never done this before. So let's just say we want to go ahead and create this. It was successfully created, but we're getting this connection failed, and if we look at this, it says it cannot connect to it because basically it cannot open the server requested by the login.

Now what we need to do is we need to edit this, and we need to do something quite important. We need to take this managed identity name, and we're going to actually update our database and make sure that this identity name is in there so it recognizes it and can connect to it. So let's go ahead and copy this, and let's come right over here. Let me make sure I have that there. You go. Let's do do crln in our Azure Data Studio. Now this right here is the most important thing, but we have to put that in a few different places, so I'm going to write out all the code, and then I'll explain it to you, and I'll have it to where you can just copy and paste it yourself. You don't have to write it all out. All right, so I went ahead and wrote everything out. What we're doing is we're creating a user, and that's going to be our user for a factory that we have right here. So we're adding that, and then we're altering the rule to make them a member so that we can get access. Now if we run just this—let's go ahead and run this—you'll notice that we don't have anything in this database principles and the database role members. This is our, uh, CIS databases, so we're just checking there isn't, uh, that account in there. So what we now need to run is this top part, and again, I'll have this as a copy and paste down below. Let's go ahead and run this, and looks like it worked. And now let's check these two queries again, and there we are. So now we are in both of these, uh, Cy databases or Cy tables that we need to be in, so we should be good to go. Let's go ahead and go back, and we're going to come right in here, and now that we have all that connected, let's make sure—let's see—let's make sure everything's good in here. Let's go ahead and test this connection, and there we go. You can see that the connection is successful. Now what we can do is cancel out of here. Is now this should be working. Now what we need to do next is select what data we actually want in the output. Let's go ahead and select our products. We'll select "Next." You can preview the data if you want, but we don't need to do any of that right now. Let's go ahead and select "Next."

Now where are we going to place this data? Because we have data sitting over here in Azure Data Studio, right in here, that we want, and let's say we want to put it in the Azure Blob Storage. Maybe this is a report or some type of query that we want to send to a client, uh, and so that's what we're going to do. To get it into the Azure Blob Storage, we need to select our subscription and our account storage name, and we'll go down here to "Create." Next, we have to choose our folder path. So let's pull up our storage account. We can duplicate this over here, and let's come back and let's see if we can get it right here. Here's our storage account. Let's go to Alex the analyst storage. Let's go to our storage browser, and we're going to go to our blob containers. Now we should have, uh, one blob container in here. Here we go: ACA container. So we can come back here; we're just select "Browse." I did all that just to show you where the data was coming from, but that's in our, uh, storage account over here, or is it called account Storage? Storage accounts. If we just, uh, come in here and we say, okay, we want to file, that doesn't really work because we're putting data into something, so we can't put it into a different file. We need to select a folder that we're going to place it into, so we're going to select the ATA container. We're going to select "OK." For our file name, we'll call this one SQL database output. Put. And that should be good. Let's go ahead and select "Next." Now these are file format settings. We need to specify how we actually want this data to sit once we move it into the blob storage. Now this is a delimited text. You can choose JSON files or C files, Parquet files, whichever files you want. So we're going to do a comma-delimited file, which is should be a CSV. We shouldn't need to add any compression onto it, uh, because this isn't a massive amount of data or multiple files at all, so we should be able to go ahead and select "Next." Next, we need to specify our task name. Now we're going to call this one SQL to blob, and we should be good to select "Next," and this is the whole process. So we have Azure SQL Database going to Azure Blob Storage. Let's go all the way down. We'll select "Next," and this is, uh, creating this. It says our whole deployment is complete. It validated the copy runtime environment, created the datasets, created the pipelines, and ran the pipelines. Go ahead and select "Finish." If we come over here to our storage, you can see that right here we have our SQL database output, and, uh, let's see if we can just click into it real quick. Let's go and download this just to see what it looks like. I'm just going to put this in downloads. We'll go ahead and save that. If we open up this file, uh, just ignore those pictures of me and my wife on the segue, um, if we open up this file, we can open it up as a—let's just open up a notepad—and you can see here's our data. Now this isn't in a CSV format, um, and that's okay because it's really easy to change it, but it is CSV separated, so is a text file. It meant to be—I kind of want it to be a CSV—but we can very easily, uh, change that. If we do CSV here, and now we've changed it into CSV. Let's open it up, and there we go. So, uh, you can definitely change that, and we may actually look at that at some point in this lesson, but very easy to change it to a CSV file because it is comma separated, um, so—oh, this looks great; this looks really, really good. I don't need to save this. Let's go ahead and go back. Now let's go back to Azure Data Factory. One thing I want to point out really quickly, uh, before we get into some other things, is, uh, for our recent resources, we have our SQL to blob. You can find that right over here in this author. So when we go over to author, you can find our pipelines, and we have some datasets as well for our destination and our source dataset, but that's regardless of what we're looking at in our pipeline. We built our SQL to blob, so if we come over here, this is what it's doing, and we can click on this, and we can see, uh, kind of what it's doing. Down here we have our source data; that's where the data is coming from; that's our table. You can also have it write a query, so you can change this to a write a query from that table if you want to do some advanced stuff, um, you know, this data is super simple, but if we wanted to just select units in stock where it's greater than 100, right, you can just write the query in here, copy it, and put it right in here, and then you can have a query instead of pulling over the whole table. So that's actually, uh, really, really useful. And so that is all really interesting stuff. Now within what we're looking at right here, we'll take a look at that in just a second, um, but what we're going to do is we're going to come over here to transform data. Let's go ahead and select this. You can see right up here we have our SQL to blob; that's our pipeline, and we have a bunch of stuff, and then we have our data flow right down here. Now what our data flow is is we're able to take different data sources, different things, transform it, join it, uh, do aggregations, do anything we want to it, and then spit it out wherever we want. If we want to pull all these different files and put into a SQL database, we can do that. If we want to take a bunch of different data, and we want to put it into a file, put it into blob storage, we can do that. Or if we want to take multiple tables from a SQL database and then put it into a file, we can do that. And so it's kind of limitless what you can do within here. You just have to kind of know how it works and, and kind of piece everything together. So let's start building this transformation. Let's come in here, and we're going to add a data source. Now within our data source, we can come down here, and we need to select a dataset. Now these are two datasets that we've already used; these were in the ones that we did when we created the SQL to blob. Let's say we want to create a new data source. If we select a new data source, you'll notice all the different options for this, right? We have a ton of, ton of, ton of different places and applications that we can pull in, um, if you just want to scroll down, there's a lot. And so there's a lot of different things. Now we're on a free tier, so, um, we may not be able to use some of these, but you know, if your company's paying for it, you should be able to do it. So let's just say we're going to take it from Azure Blob Storage. We're going to come right here; ours is, uh, a delimited text. Let's go to our link service and go to our storage, and we need to select our data. So let's go to ATA container. Let's take the CSV file .csv. So we just came in here; we're just selecting our data source, and you'll see we select it from our blob storage, and there's the path to it. Let's go ahead and select "OK," and there we go. Now one thing we will want to do as we're going throughout this process is to turn on data preview, so you have to turn on the debug mode. Debug mode is right up here, so data flow debug, and you can let it live for a certain amount of time. This does cost money; it's very cheap, but, uh, it's well worth it because as you're doing all these different things, you're going to want to preview the data and see what data looks like as a final result before you push it somewhere, before you put it into a database or a file. So we're just going to wait for just a second while that, um, while that debug session starts, and then we'll have our data preview available. Now this is taking forever, and I don't want to wait around forever to keep going, uh, it should work at some point, and maybe we'll see it throughout the process, but, uh, we're just going to keep going because if yours is taking as long as mine, we don't need to wait on it because it's not vital. You don't have to have it, um, so let's come down here. What we can do is we can add another source if we want to. You don't have to, but if you have multiple sources, you can add that, uh, so if you wanted to add a source down here, you just add, uh, another source. We aren't going to be doing that, so we can just delete that. Let's click this little button right here because they—we're getting a little different options than we have been, uh, before. We have a bunch of things that we can do to this data. So let's just scroll down really quickly. You should notice, and you should be able to see a ton of these, and one of the most important is actually this destination, sync destination at the end, and we'll take a look at in a bit, but we have all of these options to actually change and transform our data. And so if we want to clean the data, if we want to filter the data, if we want to aggregate the data, if we want to pivot the data, there's so many different things that you can do to it. Now really quick, uh, while we're here, I'm just going to show you the actual data that we're working with. If we open up this file, we have this CSV file. Let's go ahead and open it, and this is the data that is actually in that file that we upload, and there's a lot—there's about 32,000 rows of data. We have things like state name, county, city, place type, etc., and there's a ton of data in here. Let's say we just want to filter this data. So let's go back. Let's go ahead to—don't save; I just want to show you what it looked like. Let's go ahead and let's say we want to filter our data. So we're going to come down here; we're going to go to a row modifier, which is filter. And so what we need to do is come down here, and I highly recommend opening up the Expression Builder because what we can do is we can say we want to filter on one of these columns. Now over here in the data flow Expression Builder, we have our expression up here, and we have all of our elements. So we have things like functions, input schemas, parameters, cache lookups, data flow libraries. 99% of the time you're going to be using functions and your input schema. So our input schema is our table, and the functions are all the functions that we have in here. So we can do one—there's one called Equals—let's do this one right here. So we can do equals. You have your expression; you can even sit over here. Here's some examples that you can do, but what we're going to do is we're going to go over here, input schema, and we'll get rid of this. We have something called a state name. So if we come up here, select in there, select state name. So if we do state name, and then in here, let's do Alabama. So what we're saying is is we're going to filter where the state name is equal to Alabama. That's what we're going to do. So we're going to save and finish this, and that should be good to go. Let's see if we have a data preview. This hasn't been working for me at all. I hope it's working for you. It's just taking it a normally long time, maybe because they're not prioritizing, uh, me as a small account, which is understandable, but, um, you know, thanks Azure. So now we have this source. We filtered our data, and now—and you can do so many different things in here; we're just—this isn't a full, um, you know, lesson on how to use this. Now we need something called a sink. Now what the sink is is how you can actually save and publish it because you can't publish it without a sink. The sink is how you can specify where you're placing this data. So what we're going to do is we can place this anywhere we want. We can place this as a new file; we can put this in our SQL, uh, database. Now if we come in here, we're going to have several different options here, but let's go ahead and select a new one, and, and let's say we want to put it back in the blob storage. So we're cleaning the data up, and we're going to save it as a delimited text file. Let's go ahead and select "Continue." So now we want to put it in our blob storage again. We need to select our container. Container is going to be ATA container. You could set up another container if you want. Now I don't think we want to select a file name because, uh, we're choosing the file path where we're going to place it, so I think we're just need to select "OK" here. Now as last thing that we want to do before we actually publish all this, let's go to settings, and we have some options in here, which is a file name option. Now when you're doing files in here, you—in fact, you should probably publish this and just try it and see what happens—but oftentimes you're going to want this as a single file, but that doesn't always happen when you're using Azure. Sometimes it'll make it two, three, four, five, six files depending on what you're doing, so I like to output this as a single file, and then we get to output—put as a single file—and we can select the name. So we can just put this as—we'll do filtered data—and that's it. I'm actually not sure if I need to do CSV right here. Let's just try it as CSV. Let's go ahead and try to publish this. It looks like it says the file name option, output to single file, requires single partition to be selected in the partition type. So if we come over here, I think it's in optimize for the partition option. You can use current partitioning. We need to select single partitioning. It now says it is fixed. Can close that, and let's go ahead and publish all. So what we are going to do is we're going to review this. We have our different datasets, um, and then we have our data flow. So our data flow, we're going to go ahead and publish these. These are all changes that we've made, uh, throughout this process. It's going to deploy those, and then it's going to, uh, run this data flow one, and then we should see a new dataset, uh, which is filtered based off of our source data. We should see that in our storage account. We should be able to come right up here, and it looks...

Like all those things are completed, I forgot we're actually not going to see this, uh, because it didn't actually run; it just saves as a data flow. We need to come up here, and we need to create a new pipeline. So let's come up here, create a new Pipeline. And what we're going to do is we're going to go down to our data flow; we're just going to drag this over. So this is our data flow, and we can name this anything we want. We can say, uh, transform—I need to spell that right—uh, transform data; let's just call it this. And then we need to add a trigger, so we're going to go ahead and say Trigger now, and that should run that data flow that we created. So let's go ahead and Trigger now, and actually we need to publish it first. So let's go ahead and publish all; we'll publish this, and that publishing is completed, and now we can trigger it. So now we're going to say, okay, and it's going to start running.

And so this is one of the things just to kind of understand about building data flow and pipelines is that the pipeline—you can do a lot of different things in it—but you're mostly chaining together different data flows. When you actually get in the data flow, you're chaining together all the different combinations that you want to do with different data sets and Transformations and data cleaning and all these different things, and then you put it into a pipeline. And that pipeline, when it runs—which I think it's still running—when it runs, it runs that data flow that you created. And so these data flows can get really complex, and you can chain multiple data flows together. You can say, once this data flow is done running, then do this; once this data flow is done running, do this next data flow. And so there's a lot of different things that you can chain together, and it's pretty awesome. So we're just going to wait on this for just a second, and when it is complete, we'll take a look at the output. And there you can see that it succeeded. Let's come over here and take a look; let's go ahead and refresh this; let's go to our blob container, and there we go; we have our filtered data.CSV. So you can go ahead and check that out. Uh, so those two things are working great. Let's go back home really quick.

So we've worked on ingestion; we worked on data transformation; and lastly, we have this orchestrate. And let's just come in here really quickly and take a look at how this looks. So as I was telling you before, we have these, uh, different pipelines that we've created; we've created a few extras that don't do anything, um, but what we can do is within this, we can orchestrate this by taking multiple different Pipelines, multiple workflows, multiple data sets, and orchestrating all of it into an actual full pipeline. Now with that—and we're not actually going to be, uh, doing that right now—but with that, you can see that we have a ton of options right here. And in fact, you can take data from a ton of different places; you can copy data; you can do a spark job function or even an Azure function; and you can customize so many different things and take data from so many different places, and you can use so many resources within Azure to do just about anything you want. And so, uh—and I don't say that, you know, as in like you can actually do anything—but just look at all these options; there are so many things, uh, that you can do.

And so I've just given you a brief kind of introduction to ingesting data, uh, and transforming data, and then to build these out, you just kind of combine those things together. You say, Okay, I want this, uh, pipeline right here, and then if that works, then we want to, you know, transfer it over here. And actually, we need to do something like this on success, and so you can say, if this works, then on the success of that, we'll do this, and then just for an example, we can say, um, we'll add this down here; we'll say if it fails, we'll do this one. And so this is just a demonstration of how this looks; this isn't actually what you should do by any means, um, but you can give it some different instructions. You can say, if this successfully works, if this pipeline works, then go do this piece, and then if this works, go do this piece. Often times you're going to chain these together, and this is a full orchestration of Azure Data Factory that we're not looking at in this lesson, but you can imagine, uh, you're taking data from a client, so the client drops data into a specific location; you ingest that data, so you have a data ingestion data flow. Once that data gets in, then maybe you have one for transforming your data, so if the data adjusts properly, then you're going to come up here, and you're going to transform the data; then you'll have another one afterwards; you'll be over here, and you'll say, once it's transformed, and then we're going to send an email to me, who is the person who own own that, saying, hey, this ingestion process worked or this data pipeline worked. And so you can get really advanced; you can also keep it really simple. And in my time as a data analyst, I worked with a ton of data engineers and data scientists and database developers who use this all the time, so I get in there, and I get to mess around with it quite a bit. And oftentimes it was mostly keeping it kind of simple: you're ingesting data, transforming the data, and you're placing it in somewhere. There were some use cases where we did a lot more advanced stuff, but this is the meat and potatoes of it. So play around with this, mess around with it, try to get these different things to work, try to create a full end-to-end project; I think that'd be really cool, and maybe I'll do that in a future video.

So that is, in general, what ADF kind of is; that's the meat and potatoes of Azure Data Factory, and I hope you're able to follow along and really understand that so that you can start building on top of that and trying out your own stuff. If you found that helpful, be sure to check out my full Azure and AWS course on analystbuilder.com. And if you haven't already, be sure to like and subscribe below, and I will see you in the next [Music] video. What's going on, everybody? Welcome back to another video. Today we're going to be taking a look at Azure Synapse Analytics in [Music] Azure. Now Azure Synapse Analytics is meant to be this all-in-one place to go for all your data analyst needs; it's going to have ET; it's going to have workflows; it's going to be able to query data, so it has a lot going for it. So we're going to be taking a look at that, and I'll talk a little bit about the comparisons between Azure Synapse Analytics and a few of the other resources and tools within Azure. So with all that being said, let's jump on my screen and take a look.

All right, so let's go down to our analytics tab; let's go over to Azure Synapse Analytics, and let's create a synapse workspace. As you know, we need to select our subscription and our Resource Group, and we need to name this, so we'll call this Alex the analyst, and this is called, uh, oh wait, I need to make them all lowercase, so we'll do Alex the analyst ASA Azure Synapse Analytics; that should be good. Now the next thing that we need to do is select a Data Lake Storage Gen 2. When we are using Azure Synapse Analytics, we are using a data link, so we need to create an account name for this, so we'll do Alex the analyst, uh, ASA; that should work. Then we need a file system. Now we don't have one, so we're going to create the Alex the analyst fs, and let's select okay. Now it says right here, assign myself the storage blob data contributor role for the Data Lake Storage Gen 2 account to interactively query it in the workspace. So we're making ourselves the person who is the main contributor so we can actually use this Data Lake Gen 2 that we're using. Let's go ahead to review and create; it is going to cost us because we're using a serverless SQL per terabit; it's going to cost about five USD for that, and that's not a big, uh, cost for us, so we're going to go ahead and create that, and that should be covered under your 200; if you have the free $200, uh, credit, that should be covered under that as well.

All right, so that was deployed, uh, successfully. If you look in here, we did a few different things, or it did a few different things: one, it created a new storage account for us that's going to be our Data Lake; next, it created our Synapse workspace, so we have the new storage account, and we have a new workspace. Let's go to this Resource Group here; let's actually come into our Synapse workspace, and you can see a lot of our information up here, but then the most important part, which is open Synapse Studio. So let's go ahead and open this up; then we need to go ahead and sign in here. Now let's just take a look at some of the things that's in here because this should seem pretty reminiscent of Azure Data Factory, and I say that because in our last lesson we looked at Azure Data Factory, and we had a lot of these different things; we had things where we can ingest data; we can transform data; and then we can place it somewhere. And believe it or not, you're able to do that within Azure Synapse Analytics as well. Not only that, within Azure Synapse Analytics, we're going to be able to explore our data and analyze it as well. You can come in here and see how all that works, but this is going to act very much like an Azure Data Studio. If you remember from our SQL databases video, we looked at Azure Data Studio and how that works and how you can actually access the data, and we'll be coming back to, uh, some of these things in a little bit. So we're going to keep this up; let's come over here, and let's take a look at our data. Right now we have no data in here, so let's come in here. Now if you connect to a SQL database, we're actually creating a SQL database or a lake database, but we don't need that; we're going to connect to external data in here; we can connect to our Azure Blob storage; just want us to go ahead and say continue, and we're just going to bring in some of our data that we've been working with. So we're going to go to The Alexan list storage; is we have some, uh, data in here, and we need to select our subscription right there; there we go; we we'll go ahead and create that.

So now if we come over here in our workspace, if we go over here to linked, we have our, uh, Blob storage; then we have our data link. Now within our data link, uh, we don't have any, uh, data, but if we come up here to The Blob storage, and so as you can see here, we have all of our data that we have in that Blob storage, and if we wanted to access one of those, we can do something like select new script; we do a new notebook where we can load to a data frame or a spark table; we can create a data flow; we can take that as a source; we can transform the data; and we can load the data to somewhere else. So there's a lot of things that we can do with this data. One other thing I want to show you just with the data section is we have these browse Galleries, and within here there's a ton of real and really unique data that we can, uh, use, and we can look at. For example, let's come over here to New York City, uh, safety data; let's go ahead and click continue; there's a bunch of data in here; we're going to add this data set; it's going to take just a second, but it looks like it was created in this sample data sets, uh, and Azure Blob storage. If we come over here, we can click on City Safety New York; let's go ahead and right-click on it; select top 100 rows, and we'll go ahead and run this. As you can see down below, we're just looking at, you know, data, so this is how we can run it with SQL, and with this, we can do just about anything we want, right? If we wanted to do some type of group buy or joins or anything we want to do, you can do it; this is going to be by default SQL Server, um, so you can do a lot of things, uh, that you want to do in there. Let's just say we wanted to group by category, um, and subcategory, and then up here we'll select the top; we'll just do all of it, but we'll do select category and subcategory, and then we'll just do—let's do a count of the subcategory; we're just going to get a count of it; let's go ahead and run this, and let's take a look. Okay, it looks like our output is correct; I was getting worried at first when I saw the nulls at the beginning, but we could probably, uh, clean this up a little bit, but we don't need to. Okay, we're just taking a look at Azure cups analytics can do. So this is where we can query data; we can also save this; we can publish this, and now that's been saved. So if I came over here to the develop, you can see we have SQL script one. Now we just got to change the name of this; if we want to change this something else, we can come over here; we can say rename; we can say this is the, uh—was this New York City?—we say Group by, uh, category; we'll just select okay, and there we go. So now we've saved that script, and so that's really important; that's really, really useful.

One other thing that we can do down here is we can also look at this as a chart. Now this data in and of itself isn't very helpful as a line chart, but if we did a bar chart for the category column—was the category—and maybe we just want to select the top 10; we'll order by—let's do the count of subcategory—let's try running this again, and we can go look at the table really quickly, and I need to do this descending; whoops. All right, let's run this again. All right, so there's the data that we want; let's go back to this chart; again, this isn't a good visualization; we would do something like a bar chart; we would have the category here, and that'll be fine, but you can customize this a little bit more if you'd like to, um, but we're able to look at little visualizations in our output when we're working with these SQL scripts. If we want to, we can save that as an image, and that would be great. Again, we can publish this, and we can save this, and we are good to go. Now the next thing that we can do—let's go back to our data—next thing we can do is we can also create a notebook. So let's load this into a data frame, and you're going to notice this looks a, a little bit different. Now before we can actually run this, we have to create or attach a spark pool; we don't have one, so we're going to go create one, uh, really quickly, and we'll just call this, uh, ATA pool; there we go. We're going to go back down to review and create; going to create this; that was successfully deployed, so our, uh, pool is ready to go. So let's go back here; we're going to go to our ATA pool, and we're going to choose pySpark; that's the one I've almost always used; you can also use SQL in here now, um, SQL in, uh, Azure SS analytics or in other tools maybe like Data Bricks and other things like that; they don't always have a SQL option, but now it's being added a lot more, but I used to have to use Scala and pySpark and all sorts of things, but now if you know SQL, you can do that in here too, um, so you're good to go. We're going—this is pySpark though—so we're going to go ahead and run this, and right here we're running into an issue: your spark job requested 24 V cores; however, the workspace has a 12 core limit, and and that's because we need to message Azure support and request more, um, and unfortunately for this video, we're not going to do that. So let's get a smaller data set, so we're going to come back here to our storage, and let's add a new file; it's going to be this products file right here; let's go ahead and upload this, and there we go. Now we can go back here; we're going to take a look within our container, and we have this products.CSV. So let's right-click on this; let's go to new notebook; you can load to data frame or spark table; whichever one; we're going to load this to a spark table; let's choose our pool, and we're using Python, so now let's go ahead and try to run this. Now this didn't work; it says these are the cores on the memory of the driver excuses exceeded the spark pool node size. So for this, what we're going to do is we're going to come back here; we're going to edit this, and we're going to make it just the smallest one possible because we're just exceeding, uh, what we are working with right here. So now that that's updated, let's go back; let's come right here. All right, and we got our output; took a while, but that's cuz we're using a really small pool over here. Now if you use a larger pool, it'll go a lot faster, uh, so if you select a ton more nodes and, uh, you get a larger node size just in general, it's going to go a lot faster. And so right now—oops, let's get out of here—right now, uh, before when we tried it with this one, it wasn't working because our node sizes were just too large for what they're allowing us to have; we have to again ask them to increase that, but again that's mostly for really large data sets; for this one, we have a really tiny data set, and we get our data right down here, and so everything is working properly. But what we can do—which is really neat because this is a notebook—is we can come down here; we can write more code. If you use something like a Jupyter Notebook, or there's tons of other notebooks out there that are really great, um, if you use a notebook, you know, you have some code up here; you can put code down here; you can also put markdown, and so you can add in, you know, any title that you want to, uh, add in here; you can say this is a title; we go ahead and run this. And another really cool thing about using a notebook, especially within here, is that you don't just have to use Python. If you, uh, for some reason you say, Okay, I want to use SQL, you can do that using this magic command, and you can say .SQL, and then you can just write SQL like normal. So you can select everything; we'll come right down here; we'll say from, and right now it's called Data frame, but I don't know if that's actually going to work; let's go ahead and run this, and that's not going to work because it's not, uh, in a table. So we can do dataFrame.Dot, and we can say create, and there's this option for create or replace Temp View, and then we can name whatever we want, so we can say this data frame to table, so we'll call this, uh, that right there; let's go ahead and run this, and then we'll come down here and replace that right here. Now we can run this, and we just created it into a temporary View, and now we can run this just like we're running SQL. And so there's lots of options that you can do with this; I have a hundred tutorials on using Jupyter Notebooks and how to use, uh, Python; it's very similar within this, and so it's really neat to be able to combine the two, uh, languages, I guess if you want to call SQL a programming language; it's technically not, um, but we have py—so where you can use Python—and we have this as well. And so lots of options, lots of really good stuff in here, and so that's a big part of working with data within Azure Synapse Analytics. Now if we come down here to

Integrate. We can create now pipelines, link connections, and copy data tools. We're not using them, but we can use these things, and this is very, very similar to Azure Data Factory. There are also some really good uh options in here if you want to kind of copy these. Let's say, right over here it says "Bolt copy from files to database." If you want to do that, you can come in here, click continue, and then you can use this. So you can open this pipeline; you can say this is what I'm looking for, and you select your link service and your other link service. You open the pipeline, and it creates that pipeline for you. So just something to be aware of. I think that's really awesome, and in fact, I've used these uh many times. Let's go ahead and click back. So there's tons and tons of just sample things where if you are doing one of these—let's say you want to create a system that deletes files older than 30 days—there you go. And so there's tons of things in here that you can use, um as well as there's different SQL scripts you can use, different notebooks, different data sets. These are all free things that you can use within Azure Synapse Analytics. But let's get out of here; let's go ahead and close this out.

So let's say we do actually want to use something and integrate. Let's uh use this copy data tool, and this should look extremely, extremely familiar if you use the Azure Data Factory uh if you took that lesson. Now we're just going to copy it from blob storage. We'll go to next; we'll go to browse, and we'll just select a file. So we're going to come in here; we have our products uh data, so we're going to take that products data; we're going to select next, and it wants to know what our file format is. Right now, we have it kind of as AC CSV, and we don't need any compressions, so we're going to go ahead and keep it as is, and we need to select our destination. Now we can put this in almost any place we want; in fact, Azure uh SQL database would be a perfectly good spot for it. We're just demonstrating how to move one thing to another, so I'm going to select Azure blob storage. We're just going to place it in the exact same one, exact same container. We're going to make some uh weird copy of this. Let's go ahead and click okay, and we need to select the file name. Now the file name is going to be products_copied. It's all we're going to do, do CSV. So now we'll select next; we'll select next, and our task name is going to be copy_data; we'll do underscore, so copy_data, and we'll select next, and we're going to select next. Now I'm not going to deep dive into the transformation and the pipelines and all those things, cuz that's what we did in Azure Data Factory, and in Azure Synapse Analytics it's very, very similar. Now you can see here we have this pipeline built; we can come right over here, and this pipeline is ready to go. All we have to do is we have to publish it and then add a trigger. So let's go ahead and publish. All we're going to go ahead and publish this. Now that we've published it, we can add this trigger. We're going to say trigger now; we're going to click okay. So as we looked at in Azure Data Factory, once we publish it, it saves it; we can add this trigger; we can run it to trigger it now to actually activate that process, and now it's going to copy that data over. Now that's should take a little bit of time. We can view that pipeline running, and it looks like it just succeeded. We can go back to our storage account; we're going to refresh our storage uh really quickly, go back to our blob containers, and there you can see we have products_copied.csv.

So even within Azure Synapse Analytics, we can create these workflows and these pipelines. So let's go back over here, back to the integrate. So this integrate allows us to do just about almost everything that we were doing within Azure Data Factory. It is a little bit more limiting, but we copied some data; that's what we worked with, and there's a bunch of other things in here as well. You can create full data pipelines within Azure Synapse Analytics, just like you can in Azure Data Factory, but within Azure Synapse Analytics we can also use the data. So we can take a look at the data, and we can uh run queries on it, and we can create different notebooks as well, and this is really awesome stuff that we're able to do. So Azure Synapse Analytics is really meant for a more mature data team; they're using large amounts of data. Again, this sits on top of a data lake, so you're creating all these pipelines, you're working with all these different data sets, and you're querying the data; you're using the data; you're transforming the data; that's what this is really, really for. And so if you come back over here to the home, you can see that we ingested some data; we explored and analyzed some data; we also have visualized data. Now if we click in here, we can connect to Power BI. We don't have an actual workspace for Power BI set up, so we can't do that, but this is meant to be something that you're supposed to do; you're supposed to be able to ingest it, explore it, and then visualize it all in one place. So it makes the work for data analysts or even data scientists quite easy. If you have kind of a smaller team, or depending on your team, what data and tools you use, you may be able to do most of your work within Azure Synapse Analytics. Again, just a depending on uh, you know, how everything's configured, but if you have a lot more complex systems, a lot larger data sets coming from a lot of different data sources, you may need to go branch out and use some of these other tools like SQL databases; Azure Data Factory has standalone tools, but sometimes you're able to get it all within Azure Synapse Analytics—all your ingestion, your data analysis, and your data visualization all in one place. And so this is a really powerful tool, and this is one that I used for quite a while. And so uh just coming in here, this is really bringing me back to one of my previous data analyst jobs where we were in here all the time.

Now one thing to note, and I didn't really mention this, is that this is actually a workspace, and so uh we have this workspace that you can then share, and you can collaborate with people on. And so if you're over here in a notebook and you're trying to run it, it's not working, or you know whatever it is, um we can run these, run this one real quick, it'll take a second, um we can run these, and you can collaborate with other people, and they can see your code. So then you can, you know, publish this, and you can save this, and all these different things, and other people can view this. And so it's kind of more of a collaborative environment as well, which is really nice. So get in here, mess around with it a little bit. This is kind of just a little crash course on how to get up and running and some of the features of Azure Synapse Analytics, and this is a really great tool to know how to use, and I hope that that was helpful. I hope you got kind of a good overview of how Azure Synapse actually works. If you haven't already, be sure to check out my full course on Azure and AWS on analystbuilder.com. Be sure to like and subscribe below, and I will see you in the next [Music] video.

What's going on, everybody? Welcome back to another video. Today we're going to be starting our AWS series [Music]. Now AWS, to me, always seemed really complicated compared to Azure because I started with Azure and I knew it really well. And so when I started using AWS, it just seemed like a whole another world. So if you've already gone through my Azure series, that's really good because I'm going to reference it quite a bit to make some comparisons, but we still will be focusing mainly on the AWS portion of it. In this series, we'll be getting everything set up; we'll be looking at S3 buckets, Amazon Athena, Glue, Glue DataBrew, and QuickSight. And so that's a lot of different tools within AWS, and these are these are ones that I think are really, really important to know how to use. In this video, we're going to be creating an account and doing a walkthrough of the user interface. So without further ado, let's jump on my screen and get started.

All right, so what we're going to be doing is creating an AWS account, looking at the UI within AWS, just getting familiar with it before we jump into some of the tools within AWS. Now here I'll have this link down in the description; we are going to get started for free. So let's come right here, and what we want is the AWS Free Tier. With the Free Tier, we get a lot of things with AWS completely for free. Of course, there are some things that are not underneath the umbrella of the Free Tier, and so we won't be using those, but we will be able to do everything in this series with just the free account. So let's go ahead and create our free account. We need to sign up for AWS. Let's go ahead and verify this email address. So I'm going to use my AlexTheAnalystYT@gmail.com, and my account name is going to be AlexTheAnalyst; we'll do AWS. I got the code; I'm put it in here and verify this. Now we need to select our password, so I'm going to put my password in right here. There we go. Let's go ahead and click continue. Now within this Free Tier, we get a few different things. One—the free services that were within AWS—they're never going to expire; you'll get them free always, which is really great. Um, we also get 12 months free for certain services, and we'll look at that in just a little bit, and these things activate from from when our trial actually starts. So let's go ahead and fill out all this information, and then we will continue. Now if we do use any services, we have to have billing on hand in case we use something that is not free or we go above the Free Tier, and so you have to input some type of credit or debit card number just in case you do that. Within what we're doing, we shouldn't be doing that, or if we do, it'll cost like 10 cents, and so it should be super, super cheap. So go ahead and fill in your information here. Next we need to confirm our identity. I'm going to have it send me a text message so that I can uh fill this out. So go ahead and do that as well.

All right, we're going to go ahead and select continue, and now we need to sign up for a support plan. Now we're using a free account, so we don't really need support. Now if you are just you're feeling wild uh and you want to get something like developer support, you can. If you're encountering issues, um or if you're using AWS for business, maybe you want business support, uh you know what, go for it; I'm not going to stop you, but we're going to be using uh the free support because I don't want to pay for it. So let's go ahead and complete our sign-up, and just like that, we have created our AWS account. Let's go to the AWS Management Console. Now we actually have to sign in with our account. Now we are a root user, so we're going to come in here; we're going to put in our email address and sign in. All right, we are all signed in. Let's go ahead and click next, done, and done, and done, and get rid of all this stuff. Now this is our console home. You'll notice that it's very, very blank. We we don't have anything that we've recently visited; we don't have any applications running; we don't have any cost uh going either. And so once we actually start using some of these services, when we get into Glue and start automating things, you know, you might have a cost, but it should be under the Free Tier, but you will have a cost when we start using databases and instances and all these different things. These things are compute and resources that AWS offers, and so we'll be able to see and monitor a lot of those things within this Home console right here. Here you can also customize this console home if you want to. So if you want to come in here and add widgets, you can add different metrics or different things that you want in here. We're not going to be doing that in this lesson, so let's take a look at this UI really quick. On this left-hand side, we have Services. If we come here, this is all the services that AWS offers, and I'm just going to kind of slowly scroll down. We have compute, containers, storage, databases, machine learning. Let's keep going down a little bit. We have Analytics. For this series, we're going to be focusing on things in this Analytics tab, so right in here. We're also going to be looking at some stuff in the Databases, and then of course S3 for S3 buckets. This is where you can access all the resources and all the services within AWS. Another thing to note is this right up here, which is our region. Now I'm in US East, but make sure you have the appropriate one that you're using. The region actually is uh pretty important. If you choose one that's really far away, you're going to have some latency and some delays on retrieving data or using different services, and so make sure this is the correct one for you. Lastly, if we come over here, you can notice in your account we have account, organization, service quotas, billing, cost management, and security credentials. The two that I think are really important is Account and Billing and Cost Management. If we go over to Billing, this is an actual service uh within AWS; you can look at all your costs. And so so as you start using these different services, you're going to want to come in here and make sure you're not, you know, spending too much money. Just as an example, we use AWS for Analyst Builder, and so in here we track all our costs, all our bills, everything associated with AWS for our platform. And so in here we track a lot of stuff, and we have different metrics and different flags that we uh have in case we go over a certain amount or if something isn't working. And so we monitor uh you can create monitoring stuff; you can monitor a lot of stuff in this Billing and Cost Management home uh which also was really important when I was a manager and we were using AWS. And then lastly, of course, we have our Account. Within here, there's a ton of stuff like bills, payments, you know, if you want to change your credit card and all these different things. So this is a really good place to come and just be familiar with. So let's just go back to the services real quick while we close out, and we'll go to All Services and view all services. So this is what we're going to be focusing on in the next several lessons. We're going to be looking at things like S3 storage. Let's come down here; I think we're going to be looking at Athena, AWS Glue, DataBrew, AWS Glue, and a few others as well. So we're going to be in here; we're going to be learning a ton of stuff about AWS, and hopefully by the end of the series you'll be really familiar with AWS; you feel really confident putting it on your resume and actually knowing how to use it. So I hope that this was helpful, getting everything set up. If you have not already, be sure to check out my full AWS and Azure course on analystbuilder.com, and if you like this video, be sure to like and subscribe. I will see you in the next video [Music].

What's going on, everybody? Welcome back to another video. Today we're going to be taking a look at S3 buckets in [Music] AWS. Now S3 buckets are super flexible ways of storing your data within AWS, and they kind of connect to all the other data aspects of AWS as well. So if you have data sitting in an S3 bucket, you can connect it to a SQL database; you can connect it to a data visualization tool or an ETL tool or a ton of other things. So we're going to be diving into how to set up your S3 bucket, how you can actually use it; we're going to upload some data, and we'll talk about little tips and little nuances of using S3 buckets that I've learned over the years. So with that being said, let's jump onto my screen and take a look.

All right, so let's get started by coming into our services. We're going to go down to Storage; we're going to click on S3. Now you'll notice we have S3 Glacier over here, and I'll briefly mention this uh a little bit in this video because it is worth noting, but let's come over here to S3. So this is kind of the homepage for Amazon S3. We'll take a look at a few things in here really quickly. They do have this little video here, which if you're just using S3, you should look at. Now you may be wondering what does S3 stand for? It stands for Simple Storage Service. It's meant to be a really simple way to store just about anything in the cloud. You can store essentially any type of file, whether it's structured, semi-structured, unstructured; it could be almost anything. And so let's take a look at some of the things in here. It says, "Store and retrieve any amount of data from anywhere. Amazon S3 is an object storage service that offers industry-leading scalability, data availability, security, and performance." Let's come down here and take a look at some of the benefits and features. You can read through all of these things, and they're all really good. I'll read through this one in a little bit, but it says, "Data performance and durability, security, compliance and auditing, granular data control, and flexible storage options." So right down here it says, "Save cost without sacrificing performance. Store data across a wide range of cost-effective storage classes and support different data access levels that are all designed for specific use cases." Now in this lesson, we're going to be taking a look at some of this, which is their storage classes. These are quite important, and they do affect the cost that it's going to take in order to store your data, because uh it's not free; storing your data in the cloud is not free. You can also look at some of their use cases as well as some of their case studies if you would like.

Now let's actually get into it. We're going to come in here, and we're going to create our very first S3 bucket. So click on Create bucket. The first thing that we need to do is do some of our general configuration. We're going to keep this a general purpose, and we need to give it a name. So I'm just going to call this one AlexTheAnalystBucket, and if we had a pre-existing bucket, we've already configured it with uh different configurations, we can just select that bucket, and it'll copy over all the uh things that we want. The next thing we need to do is you need to come right down here for object ownership. You can have ACLs disabled or ACLs enabled. If we go with the recommended route, it means that all the objects in this bucket are owned by this account—your account that you created—but if you do it where it's enabled, it says, "Objects in this bucket can be owned by other AWS accounts." So with this one, it's just a little bit more secure because you're not saying other people can own it, which means they can delete it or change it in any way. Next we're going to come down here to block public access for this bucket. Now this part is actually very interesting because I've had a lot of use cases where you just want to block all public access, but if you start really getting into AWS and you start using a bunch of different tools and things, sometimes you need to get rid of it, um and you need to come in, and not only that, there's much more advanced things in order to grant different bucket policies or create different bucket policies, and um we most likely won't get into all that in this lesson, but it can get quite advanced. And so this piece is pretty deceivingly simple, um but we're just going to keep it as we block all public access, but as you uh start opening it up to different services within AWS, you may

Want to turn this off so that you can have different Services hitting off of your S3 bucket or the data within your S3 bucket? So we're just going to keep this as all public access off. I'm just giving you lots of extra information while we're in here, some of my thoughts.

Next, we're going to do bucket versioning. We don't need to have any type of versioning or Version Control within our bucket. What this means is keeping multiple variants of an object in the same bucket. It's used to preserve, retrieve, and restore every version of every object stored in your S3 bucket, and that is, of course, going to cost a little bit extra, so we don't uh need that at all.

Next, you can add tags, and tags are helpful if you have a lot of different buckets and you maybe—it's per client—you have some type of tag for a specific client. Perfectly normal.

Next, we have default encryption. Now, encryption in general is really interesting within the Cloud. I've run into lots of use cases where it's been really difficult to work with. If you want to pull data into certain Services, you have to decrypt it uh because you've encrypted it in one area. And so if it's in the S3 bucket and it's encrypted, uh especially with kind of more advanced options, it can be somewhat difficult. And so just something to take into consideration if you want to kind of up your encryption, but for most use cases, you're just going to use the server-side encryption with Amazon S3 managed keys with the bucket key enabled.

Next, let's come down here to advanced settings. There's only one thing in here that we need to look at, which is the object lock. And remember, everything that you put inside of an S3 bucket is an object. Now, if we just read this right here, it says storage objects use a write once read many, which is a WORM model, and this helps prevent objects from being deleted or overwritten for a fixed amount of time or indefinitely. So if you want to drop a file in there and you know you're going to need it, you don't want it to be deleted for any amount of time, you can lock that, and you can have an object lock on that object; it can never be deleted. We, of course, do not need that, so we're going to keep that disabled.

And now we're ready to create our very first bucket. I clicked on create bucket, uh looks like we can't use uppercase, and I actually knew that. Uh, let's fix this. Now this does have to be unique; this is global, so it says right here you have to use a unique name; you can't just use like Alex—someone's probably already chosen that. Let's go ahead and create our bucket, and now you can see right in here we have our very first bucket—very exciting. Let's click into our first bucket, our only bucket. Let's go ahead and click into here, and we have a few different things in here: we have our objects, and that's going to be any files that we upload into this bucket. And within this, we can create tons of folders and subfolders and sub-subfolders and all these different things. We also have properties, so this gives you a little bit of information on how you actually created this. You have permissions, so if you want to Grant uh access to it, I talked a little bit about this, which is uh bucket policy. So if we turn off this all public access and we turn on this or we allow this bucket policy, we can create our own bucket policy, and that's written in JSON, and we can create our own bucket policy for public access to this specific bucket. That gets a little bit more advanced, but it is really fun. I've uh done that quite a bit. You have metrics on this bucket; you can manage this bucket, and you can create access points to this bucket. So there's a lot of things just within a single S3 bucket that you can do, but the most popular one that you're going to be using is this one right here, which is just adding, creating, and deleting and using objects in general.

So let's go ahead and upload our very first file. We're going to go over here to add files, but you could add a folder if you have a whole folder, but we're just going to add one file. And within my sample files here, we have a bunch of very real Healthcare data; it's not real—it's just completely fake data. Now what we're going to do is we're going to upload one, then we're going to come back; we're upload another one in a different way, and I'm going to demonstrate that and uh explain that in a little bit. Now what we're going to do is we're going to select this file; we're going to open this up, and now we have this real Healthcare data 1.csv, and that's what we're going to be uploading. Now let's come down here; we have our destination; this is telling us where we're actually placing this; we have our permissions, and of course, we have uh bucket enforced. And so if we wanted to Grant access to other accounts, we need to change some of our access policies. But lastly, we have properties, and this piece is really interesting; this is our storage class; this is how uh Amazon S3 is actually going to store your data. Now I highly recommend going into learn more and looking at their Amazon S3 pricing because it's very fascinating how they do this, um and it's also really important that you understand the differences between these different options. By default, we have standard, and it's designed for frequently accessed Data within milliseconds for Access. So if you're going to be using this data, you're going to be hitting off of it for different applications, Services, uh visualizations, whatever you're using it for, you're going to want to be able to access that pretty quickly. And so this is a great option. If you need it even faster, you have S3 Express One Zone, which is single-digit millisec response times for the most frequent access data. Now I don't think it talks about cost in here, uh but you can use a calculator, and this is going to be costly, right? It's going to cost more to get faster and lower latency responses to your data.

Now if we come down here, you'll notice we have this Glacier tier. Now if we went back to our resources, remember we talked S3 Glacier; I was going to mention that. Well, you can use these different glaciers, which allows you to store it for long periods of time at a much lower cost, but it stores it a little bit differently. This one specifically is instant retrieval; it's very similar to almost standard, but you're kind of storing it for a long, long time. You may not need it for 10 years, but when you do need it, you need it right away. Which is not a lot of use cases if you're using Glacier, but then you have something like Glacier Deep Archive. This is long, long-lived archived data; access less than once a year with retrieval hours. And so this is going to be data that you don't need it right away; uh you just want to be able to store it and have the security of putting it in the cloud, but you may not use that for years. And when you do need it, you know you're just—you're okay with waiting a little bit. It's going to cost almost nothing to store; it's very, very, very little. And oftentimes if you're doing something like this, you may even put backups of databases; you may put backups of entire code bases in here, things that you may never use again, um and there's lots of different use cases as well. So I just wanted to walk through that. With this file, we're going to do standard, and on the next file we do, we're going to do Glacier Deep Archive and just look at the difference here. Let's come down—we don't need encryption; we don't need any type of checksums, tags, or metadata; that stuff is almost uh never used or very, very infrequently. So we've uploaded this file. Let's go ahead and go to our destination. You can see that now we have an object in our bucket, which is fantastic. You can even come over here, and you can see the storage class is standard. Now let's go in; we're going to upload one more, but now we're going to upload it as a different storage class. So let's go to upload; let's go to add files; let's go to healthcare data 2; we're going to open that up; the only thing we're going to change is now we're going to go down to the Deep Archive, and we're going to go ahead and upload this—easy peasy. And there we have two files now; we have one in standard, one in Glacier Deep Archive. Let's go into the first one. If we want to use this in any way, it's almost instantly retrievable; we can download this—we have a bunch of object options, so we can download it; we can copy it; we can move it; we can change the storage class; we can do a bunch of different things. But let's come back, and we're going to go to that file two. Let's go ahead in here; notice right here we're getting a totally different uh message; it says the object is stored in Glacier Deep Archive storage class; in order to access it, you must first restore it. So you have to initiate a restore right over here, and it can take 30 minutes to several hours. Notice we cannot download this; we also can't really do anything with it until we actually initiate that restore and it is restored for us to be able to access it. And that's just something that I think is worth noting about storing things in S3; there are different storage classes depending on your use case and what you're using that data for, and of course, you should look at the S3 pricing for both of these, cuz this one is going to cost more than this one. Now there are costs associated with restoring it, but if you're restoring it once per year or maybe once every five years, it's going to be significantly less than your standard storage class.

Now this is kind of the meat and potatoes of using S3. Of course, you can create folders and subfolders, and in fact, when we get into some other lessons, especially things like Glue, uh having folders, subfolders, and different things like that is actually really important because we have something called a Glue crawler, and it'll crawl through your folders and subfolders and retrieve certain data. And so having a folder structure actually becomes more important in those lessons, but we'll of course get to that when we actually start looking at it. One other thing to mention, and this is just uh kind of a neat addition that they have in here: if we go into this file, we go down to query with S3 Select, you can access this data and take a look at it. And so uh let's say we have uh our CSV here; it's comma separated; we can query this data using SQL. And so let's just run this with the limit five on there; we'll run our SQL query; we have our query results right down here; we can have it formatted, and it looks like a little table. So we can come in here, and we can look at uh this data down here and just see what's in there. And so that might be useful to you. Now in the next lesson that we're going to be having, we're going to use Amazon Athena, which is basically a tool to query off of S3 buckets; that's its primary use. And so most of the time, I honestly am not using this almost ever, but you can even see right here it says Amazon Athena; that's what we're going to be looking at in the next lesson. We'll be seeing how Amazon Athena works, how you can query data, kind of set up a pseudo database-like structure. I'll talk about the pros and cons of Amazon Athena versus other tools because Amazon Athena is not for every use case. So I hope that that was helpful; I hope you enjoyed it. If you have not, I have a full AWS and Azure course on analystbuilder.com; be sure to check it out. If you like this video, be sure to like and subscribe below, and I will see you in the next [Music] video.

What's going on everybody? Welcome back to another video. Today we're going to be taking a look at Amazon Athena in AWS [Music]. Now Athena is a tool that allows you to query data in your S3 bucket without having to put that data into some type of database; you're just querying it directly. So let's say you just got a big file from a client; you put it in an S3 bucket, but you don't want to actually create a table and take the time to do all that whole process; you just want to take a look at the data; you just want to query it really quickly. Well, Athena allows you to do that. And so if that's the case, it can save you a lot of time and money. And so this is a really great tool that a lot of companies use. We'll talk a lot more about Athena in just a second, so let's jump onto my screen and take a look.

All right, so let's come into services, and we're going to go all the way down to the analytics Tab, and we're going to go into Athena. Now before we jump into actually querying data and setting everything up, let's take a look at some of the information that they have on Amazon Athena. This right here—this start querying data instantly—is probably one of the uh biggest pieces of why you would use Amazon Athena, and I'll talk a little bit about a little later on about who this is for, why you'd want to use this versus other tools uh within AWS. But Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 and other Federated data sources using standard SQL. So primarily though, for you and I, most likely you're going to be using this just to hit off of an Amazon S3 bucket. You can even see how it works right down here; it says point to your data source; you're going to use Amazon Athena; you're going to query it, and you can analyze those results. Down here in the benefits that are right here, you have start querying now; it's powerful, cost-effective, fast; all these things are great; honestly, powerful, cost-effective, and fast could be a ton of different Tools in AWS, but this piece right here is quite unique to it in the fact that you can query it as it just sits in an S3; you don't have to bring that data over into a database and actually store it in the database; you can keep it in your S3 bucket and still query off of it. And so that is uh kind of the real use case and the real reason why you would want Amazon Athena just to look at data uh within an S3 bucket.

Now we do have two options here: query your data with Trino SQL or analyze your data using PySpark and Spark SQL. We are just going to be looking at the Trino SQL in this lesson, but of course, you can use PySpark and Spark SQL as well. Let's go ahead and launch our query editor, and this is what you should see. Now it says uh we need to edit some of our settings, and I'll explain that in just a little bit, um but over here on the left-hand side, we have our data. So we have our data source; we have our database; we have tables and Views; and then we have where we query our data; and then down here where our results will be. So there's a lot of stuff just in here, but this is um not super crazy Advanced—uh if you've used anything like MySQL or if you've used Microsoft SQL Server or anything like that, this is like a really—I don't want to say dumbed-down version; that's not super uh kind to say—but it's—it's a really simplified version of it; it's not very uh difficult to understand. You can also save your queries in recent and saved queries, and of course, we have some settings over here. Now we're going to see how we can set up our database, create a table or two, see how all that works with actually hitting off of data within our S3 bucket, and we will have to fix this piece, which is you have to set up a query result location in Amazon S3; that's for your results or your metadata, and uh that piece becomes quite important later on. So let's come down here; we have our data source is going to be our AWS Data Catalog. Now we haven't covered this in a previous lesson, but you have something called a Data Catalog, and we're going to look a lot at that actually in either the next lesson or lesson after that when we look at uh AWS Glue and Glue DataBrew, but that's how they kind of organize it. So we're not going to be messing with that uh here, but we have to choose a database, and notice we don't have any database, so what we need to do is we need to start pulling in data, and when we're pulling in that data, we'll be able to create a database. Now right here we have create; you can create a table from a data source or create with SQL. So if you want to go the old-fashioned way where you're creating a table like this, and you can, and you can specify the column, the column types or data types, the location, all these things, you can do that, uh but we're not going to be doing that; we are going to be doing it with S3 bucket data. Now we also have AWS Glue crawler. Now I'm just going to open this up really quick; we will be doing this when we get to Glue—Glue—when we start looking at Glue cuz crawlers are great; crawlers allow you to specify the data source, and it pulls it in and infers based off of the column names and and the data types that are in there; it builds a table for you, and it's very helpful, and it kind of helps automate it as well. We're not going to be doing that; you can look at crawlers here, but we're not going to be doing that in this lesson. So this is what crawlers are, and that's within AWS Glue, but we're not—we're not going to be looking at that in this lesson. So what we are going to be doing is just creating it from an S3 bucket data. Now we need to call this table something, so let's call this Healthcare—and I need to spell Health right—Healthcare data; I I'm having trouble spelling. Let's go down to the database configuration. Now we don't have a database, so we need to create a database, and this is super easy—um, this is actually the table name; let's call this um patient data because I want to call the uh the database Healthcare data, so I'll call this one Healthcare data; I spelled that much better that time. So we've specified here's what our table is going to be called; here's what our database is going to be called. Now we need to specify our data set, and this is a kind of an odd part, and you'll see that in just a little bit. Let's come in here, and we have our Alex the Analyst bucket; let's go ahead and click into it. Notice though, within this bucket, we have two data sets; I I really only want this one if I'm being honest; I just want this one, but you can't do that—not within Athena and within other parts of AWS as well; this is just a uh kind of one of those nuances uh within it; we have to specify a file path, not a file itself, so we have to choose the entire folder. Let's choose this; let's come down here to the data format. Now this data format that's in here is a CSV, so we're going to come in here with an Apache Hive, and we're going to specify that is a CSV file. Of course, we want the delimiter, which is right down here, to be a comma, so we should be good to go. But because we are not using a crawler, we have to manually enter these column details. So I'm going to come down here; we are going to pull up this file; let's zoom in a little bit; these are uh everything that we have. Now we can—there is an option in here actually; we can copy all of this, and we'll come in here to add bulk columns, and you can do it like this, and then you can put in the data type, and that's perfectly fine if you want to do that, uh but we're not going to be doing that; I'm going—going to actually pull this up; we will bring this over to the side right here, and there we go. So now

We can see it, so we have our first column. It's going to be uh, patient uncore ID. It doesn't have to be the exact same as over here in the file. Now, this file type is just numeric, so we can come in here, and we'll choose integer right here, and that's all we need to specify.

Now we have several other columns, so let's just kind of bulk place them in here. We have name, we have age, we have diagnosis, and we have treatment. Let's see if there's anything else in here. I think we have files. Yeah, one more. So let's add in files. And just to show you we can do name_ is this their full name? Yeah, full name. So we'll do full_name, and we're doing that just to demonstrate that doesn't have to copy this exactly. Um, but for the full name, this should be String or text, so we could use Char, we could use uh String. See if there's any other ones that we could use. Probably those two. We'll call this one String. For age, it should be integer as well. For diagnosis, treatment, those are both String, so we'll do String and String. Then for files, this one is integer as well.

Now we can make this a little bit larger. Now there's a bunch of different things that we can specify: different types of compr compression if you have compression. So if you're using a zip file or anything like that, you can specify uh that it's sitting in it. There is partitioning as well, and this is a little bit more advanced. This is a way that you can group specific information together, but this is not something that we need to worry about, especially with our simple data set. There also is bucketing. This is a way to bucket multiple columns together, and then it's stored in a way that when you try to retrieve it, it's retrieved a lot faster and easier. Again, this is a bit more more complicated and a little bit Advanced, so we're not going to be taking a look at that at the moment.

Let's go ahead and create this table. Now we're getting this error that says no output location provided. An output location is required either through the work group result configuration setting or as an API input. So let's go fix this really quickly. We're going to actually, I don't want to have to redo all this, so let's actually duplicate this. Let's go into our query editor, and we need to go into this work group. So let's come over here, go into our work groups, and this is our primary work group. Let's come in here, and we need to come over to edit. Now within this, we had chosen uh Athena SQL, and if we come down, we have this query result configuration. We have to specify this so that our output and our metadata is put somewhere. So let's go ahead and browse this. Now, just like when we chose our file, we cannot specify like a file to put it in; it's a specify a file path. Now we're going to specify that this is our path, but this is not the best way to do it. I'm doing this for demonstration purposes only. Uh, we'll go back and do it the right way in a little bit. So this is where we're going to put our output, our results, metadata, all that stuff. Let's go ahead and choose this, and uh you can add life cycle uh configuration as well as assign a bucket owner and some encryption, but we don't need to do that. Let's save these changes, and let's go back and let's go and create this table. So it created this for us. The query was successful. Let's refresh this, and it's giving us a default database that's actually not the one that we put it in. We put it into Healthcare. So let's go over to Healthcare, and there is our patient data. You can see we have patient ID, full name, age, diagnosis, treatment, files with the associated data types. And what we can do, whoops, what we can do is we can come right over here, and let's say we want to preview this table. So that is completed. Let's go down here, and the data looks pretty good. We have this one file uh because we have two files; it's reading in that second file uh column names, but we have some data in there, and that's really really good. Now we're limiting this to 10, and let's get rid of this. But actually uh before we do that, I want to keep it the same because before we do that, we need to go look at the ramifications of what we actually did because uh unfortunately it's not a good thing what we did.

Let's go to our S3 bucket and let's see what data is in this Alexi analyst bucket here. Now you can see we have a bunch of stuff; it's not just our two files anymore. Now we have a text file, a text file metadata, CSV; this is our output and some extra stuff as well as the metadata. And what's going to happen if we come over here and we try to query all of our data? Let's go ahead and run this. Now you'll notice that we're pulling in a lot of bad bad data. This has completely ruined our query, and this is kind of defeated the purpose of setting up using Amazon Athena. What we need is we need a separate folder location within our S3 bucket to dump all this stuff; otherwise, you'll notice it's just going to keep growing. We're just going to have more metadata, more CSV, more text files. That is not a good thing.

So here's what we're going to do. We're going to create a folder; we're going to call this one uh met data Healthcare. I think I spelled that right, and we're going to create this folder. Now I'm going to take all of these things, and I'm going to delete them. So let's go ahead and delete all these files, and we have to specify this. I'm going to do a little cheat; I'm going to copy and paste this. There we go. Let's delete our objects, and those were deleted. And now you can see we just have our uh real Healthcare data here, and we have our metadata Healthcare up here now. Now what we need to do is we have to go back; we're going to go over to our work group, and we're going to specify the new metadata folder that we're going to be placing this in. So let's go back down to settings. Actually, it's in the query result configuration. Let's go to browse; we're going to go in here, and now we're going to specify the metadata Healthcare folder. So let's go ahead and specify that. Let's save our changes, and we'll go back to our query editor. Now when we run this, let's go ahead and run this again. Now we have our output; our data is in there. Let's come up here; let's refresh this. The data is not in here anymore. Now it's in this metadata Healthcare, and that's great, right? Let's come back here and let's run this once again. Let's come down, and you're going to notice something uh peculiar is what I'll call it. What's peculiar is the fact that now we have the same metadata. Now why is this? Why is this happening? And why am I showing you this in the first place? It's because when you are specifying a file path, you are not only specifying a file path; you're specifying the folders and the subfolders. So when we come in here and we look at this bucket, if we just specify that we're pulling all the data out of this bucket, then we're pulling not just this data; we're pulling a folder within a subfolder, and so now we're pulling this data as well. So we need to, if we want to keep it all in one bucket, which we can, we need to create a new folder called our patient uncore data. So we're going to click on our file; we're going to click move, and we need to specify our destination. So we're just going to put this in the patient data. So let's go ahead and choose this, and let's move it, and we're going to close this out. Now uh you'll notice that this data is a glacier deep archive, so we cannot move this, and we cannot query it. And that's also something that I wanted to mention while we were here; I somewhat forgot this. The data that we are pulling in uh this data down here is only from one file. We don't have two files of data in here. The only data that we have that we were pulling in was from this real Healthcare data one. We were able to hit off of that data because the storage class is standard, but because when we set this up, this was a glacier deep archive, we cannot query off of that data. And so what we're going to do is I'm going to come back up here to Patient data; I'm going to go to upload, so we have more than one file in here. Let's go ahead and add a file. I'm going to specify this number two, but now we're just going to keep it as is; we're not going to oh, if we go to down to properties, we're just going to keep it a standard. So let's go ahead and upload this. There we go. Let's close this out. Now the data that we're hitting off of is right here, so we're not hitting off of this file pth path. Now we're going to be hitting off of this patient data. So let's go back. What we need to do now is we need to come back up here, and we need to create a new table. So this new one, we're going to do create a new table. This is going to be a patient data 2, and we're going to choose our Healthcare data. Now our data set is going to be different, so we're going to go into the Alex analist bucket, but now we're going to specify patient data. Now this is important because there are no folders and subfolders underneath patient data; it's just this one folder with our two files in it. When we actually run this and we query off of this data, it's going to go; all the metadata is going to go into here but will not mess up our data source, and that is very important to understand. Now oftentimes, just as you know a little bit of side note, oftentimes when you're working with this, you can either set it up like this, or sometimes people and departments will create entire separate buckets just for their metadata, so they can store all their metadata in it, and then they can often they just delete it or whatever.

So let's go back here. We're going to recreate this. We're going to go to Apache Hive, and we'll do uh CSV. Note, if you do Iceberg or Delta lake or Lake formation govern table, these ones don't have options for AC CSV anyways, so that's why we're not looking at it. Um, but for Apache Hive, you do have our CSV option, which is common to limited. Now we need to go through, and we need to specify our columns again. I'm going to skip this cuz you can just do this yourself, but we're just going to do this, and then we're going to create our table. All right, let's go down; we're going to go ahead and create this table. Query was successful. Let's go back to our Healthcare data, and let's refresh this. So now we have our patient data 2. Let's go ahead and preview this table, and this is looking good. Let's get rid of this limiter. Let's come over here; let's run this. Now you notice we have 42 results, which is because we have uh one right here; this is from file; it's actually file two, and this is file one, but I got that wrong. Uh, but we have both files in here that we're hitting off of. So within our S3 bucket, we have two files of exact data, and it's creating a union between these; it's reading in both of these files, and it's giving us our output. And of course, we have this right here which we don't want, so we can always filter that out. Uh, whoops, let's come up here; let's say uh where let's do patient uncore ID. There we go. We'll do is not no, and we'll run this, and now this looks really good. And so now we're querying off of our data, and we're not getting any of that metadata issues that we were experiencing. We come back here in our metadata; it's storing all of this, and most of this is going to be pretty useless to most people. And so often times you'll have uh some type of event or something in place to delete this because you don't want to store this long term, or most people won't. There may be some use cases too, but most of the time you're going to get rid of this. And so this is Amazon Athena. Now you can also create views if you want to tie a bunch of tables together and create a view; you can do that. And of course, this is SQL, so you can write uh, you know, SQL statements like you normally would with any SQL environment or SQL editor. And so if we want to filter, if we want to join, uh we can do a lot of those things. And so this is a fully functioning uh query editor. Now if we come down, there are some other things that we have. We can copy this data; we can download these results; we can clear this whole thing. We can also create a table from this query or a view from this query, which is really useful. We can also come up here; if we want to save one of our queries, we'll come over here to query 5, and we're going to save as, and we'll call this one uh filter on nules. It's a terrible description; we're just going to save that. So we have this filter on nules, and if we have want to reference this again, we can uh come back here, and we can pull up this query, and we'll have it available. Now if we come over here on this left-hand side, you'll notice we have a notebook editor and a notebook Explorer, so there are options to create notebooks and run notebooks as well. That's something I would create a whole another lesson on because is not as straightforward as using the SQL option. But another thing we can do, which is within jobs right here, is we have something called work workflows. Now workflows are how you can create different pipelines and different orchestrations with your data and with a bunch of different tools within AWS. This is definitely a little bit more advanced and not something that most people are going to use within Athena for a lot of orchestration data flows, data pipelines; you're going to be using glue. And so within glue, we'll be taking a look at how we can, you know, set everything like this up, but you can come in here, and you can take a look at a bunch of different ones. You can, you know, these are different options; you can execute multiple queries, query large data sets, keep data uh up to date, and you can come in here, and there's lots of different uh things that you can do, but again, we're not going to be covering it, but go ahead and take a look and see if you want to try it out. It can get a little bit complicated, a little bit complex. These are things that people do all the time in AWS, especially people like data engineers, database developers, analytics engineers, people like that. And so when we get to glue, we'll look at how to create different workflows, data pipelines, uh and all that stuff.

So with all that being said, this is the meat and potatoes of Athena. Now who is this for exactly? When I was using Amazon Athena, it was mostly to query data really quickly. A client was giving me a file, put it into an S3 bucket, so I could start querying off of it, and that was it. Uh, most of the time we didn't have a full-scale production environment over on this left-hand side here with tables and Views and databases. Um, we kept the data separate, of course, for different clients and different projects, but this was not not our primary place to go. I have consulted and worked with other teams where I come in, and they're like, hey, here's where we're trying to get; we're trying to get to this place, and I'm like, okay, show me your current, you know, setup, how you're working with your data, how your data flow is work and everything like that, and they'll come in, and they're like, we're only using Amazon Athena for querying data and creating workflows and automations. If you want to be able to scale to a much larger scale, this won't work. There are several limitations within Amazon Athena as you try to scale, as you try to add more data sources, as you try to automate all these things; it gets very difficult to do within Amazon Athena. And so that's when you need to start creating different databases like using something like RDS or another type of SQL Server where you can store your data, just not within Amazon Athena. Hitting off of S3 buckets as your main uh main way to create kind of your database is not optimal by any means. And so they might work for kind of a smaller company who's just trying to hit off some data, but it doesn't work at scale very well. So Amazon Athena is really good for getting quick insights into your data. It's really good when you have a ton of data stored in S3, and you just want to get simple insights into it, want to do simple joins, aggregations, and you're not trying to do anything too complex with it. This is a great place to go because it's actually very cheap compared to other resources and tools within AWS. This really shouldn't be used as a full-scale database, especially as your company gets larger; you run into a lot of roadblocks, a lot of issues, and I've seen that firsthand. And so it's just not something I'm going to recommend, but Amazon Athena definitely has its place, and that's why we're looking at it because you most likely, if you're using AWS, you're working as a data scientist, data analyst, business analyst; you'll use Amazon Athena. A lot of companies have it; a lot of companies will use it. And so I hope that this was helpful. I hope you're able to get up and running, understand some of the nuances, especially with the folder paths. I always found that a little bit confusing until I just used it more, but understanding how to uh put your tables in to the Healthcare in here um and get that metadata folders working as well. If you liked it, be sure to check out my full AWS and Azure course on analystbuilder.com. And if you like the video, be sure to like and subscribe, and I will see you in the next [Music] video.

What's going on, everybody? Welcome back to another video. Today we're going to be taking a look at glue and glue data Brew in [Music] AWS. Now within the glue umbrella, which includes glue data Brew, it's mostly for building pipelines and creating ETL processes. This is how you get data from one place to another. You can extract data, transform data, load the data, and do a ton of other things with it. So in this video, we're going to take a look at both glue and glue data Brew because they're slightly different, and they have different purposes. So without further Ado, let's jump on my screen and take a look.

All right, let's go down here; we're going down to analytics. Now we're going to be taking a look at both AWS glue data brew and AWS glue. Now we're going to start with data Brew because I think it's a little bit more user friendly. There's some nice animations; the UI is really uh a little bit, I would say, easier to understand. And this is often a place where you'll come for a lot of your Transformations. So let's come in here, and you can go ahead and take a look at a lot of stuff in here. On this right-hand side, we have data sets, projects, recipes, uh jobs, and we'll take a look at few of those while in here, and you can look at some of the benefits as well. And uh glue data Brew is awesome. It's very much a more visual way to prepare your data, and even says it's a visual data preparation tool. And so it's a way to visualize how your data is changing. You can see how it's changing. We are not going to just create a project; we're going to create a sample project, and this is going to give us some sample data. We're going to work with it; we're going to be cleaning it up, transforming it a little bit, and then we'll see how we can automate that as well. So let's go ahead and create our sample project. Let's go ahead and select popular baby names in 2020, and then we have to go down here and choose a role name. Now we haven't talked a lot about IM roles uh within AWS, but for certain things like glue data Brew

Glue and a few other things you need. I am access so you can say, create new IM role, and that's what we're going to do. Let's go down here, and we need to actually create, um, a new role for AWS Glue BRW service. So it's going to create it, and I'll explain that in just a little bit. We'll call this one Alex the analyst, and I'll call it Data Brew. Let's go ahead and create this project. Now this needs to set up our session; it's provisioning some computes and getting our session ready, getting our data ready, all of these things in order to use Data Brew.

But while we're in here and while we're waiting for this to uh be ready, I want to talk a little bit about the UI. So over here, we have something called a recipe. Now the recipe is for when we actually use all of these things and we apply different changes to the data set. We make any Transformations, whether we join the data, we group the data, we pivot the data. It's going to say, okay, you did this and then this and then this and then this, and you can come in here and you can edit this. And so you can say, oh, I don't want to actually group it; let me get rid of that, and that'll be really easy to do, and I'll show you that in a little bit. But then we can also save this recipe. We can just say, oh, I want to save this for a future data set, or if you want to use that recipe on, you know, if you're bringing in multiple uh data sets of the same exact data, or you have multiple uh data sets coming in that are all the same, then you may use the same recipe on all of them to transform that data consistently every single time.

Looks like our data is done, or our session is ready. But let's look at this top area. So this is where we can make all of our Transformations. We can filter, we can sort; we can come in here and we can click on the clean, and you'll notice there are tons of different options in here, a lot of ones that you should be pretty familiar with if you've ever cleaned data before, or if you've ever done any of my projects in MySQL or Excel or Tableau or uh, you know, Python and all these different ones on data cleaning. And these are a lot of really popular things in order to clean the data. So there's clean, extract; you can look at duplicates, outliers; you can merge your data; you can perform functions; you can apply different functions to your data. We also have things like Pivot, Group, Join, Union, and others, and there's just a ton in here.

Now what we're going to do is first we're going to look at our data, and then we're going to go through and we're going to do a few changes to it so you can see how the recipes work, and then we'll save our recipe. So let's come in here. This is a new data set to both of us. This is our uh baby names data. So we have count, we have gender, we have the ID, we have the name, and then the year. And so this is pretty interesting information. Let's see if it's all uh 1880. This looks like just a small sample of the data, and we can come in here and we can also uh take a look at what kind of data types they assigned to this data as well.

Now what we're going to do is something quite simple. We just want to look at the male names, and we want to take a look at the most popular male names per year. So that's going to require a bit of grouping. So we're going to have to group on both the year and the name, and then we're going have to filter based off of the gender. Now one other thing to note is that right now we're just working with a sample of the data. When we actually get to the final process and we actually create this data cleaning process, we can apply this to the entire data set, or we can apply it to the sample. So right now we're just going to be working with this sample, but we'll of course be using the full data set later on. And you can even come up here and you can look at this; you can say first n rows, last n rows, and random. Now right now we're just looking at the first n rows, the first 500 rows of our data set, and that may not be a perfect sample size. It actually may be better to pull random rows, because what if we have uh male or female or other things? Right now it just looks like we have female in our sample, and that may not be representative of the entire data set. So let's come here to random rows; let's choose 500. We're going to load this sample, and as you can see we do, we do have males in here. And so I'm really glad we did that because now we kind of have a better representation of our data, and so I think what we need to do, let's bring over our recipe. I think what we need to do, or what we're going to do is: one, I want to filter where the gender is equal to male. Then we're going to come over here, and we're going to try to find the most common name per year, or maybe we'll just Group by the year and the name and do a count on the name. I think that would be really interesting. So we're really transforming it quite a bit. So what we're going to do is we're going to filter on this gender. So we're going to go over here to filter, and we're going to go down to buy condition. Now we want to say where gender is equal to male. So you can either do contains, and we can do contains an M, or we can say it is exactly; either one of these will be perfectly fine. We're going to do that on gender right here. So we have 300 females, 200 males; we don't want the females, so we can get rid of that one. We're going to only keep the males. We can also enter a value in here, uh, but we don't need to do that. And so now this is ready to go. We're going to go down, and we can either preview changes, but we're just going to go ahead and apply this. And as you can see right here in our recipe, it says under our applied steps, we have filter values by gender. We can either edit this or we can delete this. So at any time if we want to change any of this, uh, we can do that. But as you can see in our preview, in our sample, we now only have 200 rows, and it's all filtered by males.

Now what we can do is we're going to group this data. We want to do it based off of the year and the name. So let's come up here; we're going to group, and we need to select our columns. So we're going to start with the year, and we're going to group by the year, and then we're going to select the name as well, and we'll come down here and we'll say Group by, but we also wanted, and you'll notice we also wanted to aggregate these values. So we want the count. So what we're going to do is we're going to come in here and we're going to say based off of the name, we want count of the names for that year and name. So we're going to do a count here, and so for this new column, you can call it the name count, and that'll be perfectly fine. This is uh just like an example down here, but it shouldn't be a string; this should be an integer. And so if we come down here, let's see, there should be some that have multiple, um, higher than one, at least not in our preview, but still we're working on a sample here. So let's go ahead and finish this, and now we have our data right here. Now again, this is only working off the 200 rows; we could have 100,000, 50,000 uh in our data, but we grouped off of the year, and then we grouped off of the name, and then we got a count of the name. Now what we need to do is I also want to filter or sorry, sort on this year, and we can do that uh just by sorting here manually. So we can come over here, and it's just going to do this within our uh window, but I actually want to apply a sort to here. So I want to do this ascending, so the smallest year to the largest year. We'll do this based off of the year; then we'll come down here, we'll click apply, and now we have our years over here in ascending, from smallest all the way down to largest.

Just so now that we've applied all of our steps and really transformed our data on our sample data set, let's go ahead and save this recipe. We're going to go ahead and publish this. So you can add some notes if you would like. We're going to publish this, and so now we've saved that recipe. And in fact, if we come over here to recipes, you can see that we've saved this, so we have that available. And If we go back to our projects, we can come up here and we can create this job. So we're going to call this one the uh name aggregator; this is the data set associated with it, and we can choose our output. So where do we want this output to be? Let's put it in S3; uh, we can keep it right here. Let's go to our S3 location, and let's just put it in the Alex the analyst bucket. Let's select that, and we have some additional options down here. So we have some advanced settings: one, the maximum number of units. So that's to do with how many nodes you want on this job when it actually runs. You can specify if it times out or how many times you want it to retry it. This one is probably the more important one is if you want to schedule this right here. So you can come in here and you can create a new schedule, and you can specify, I want this recurring every, and this says 1 hour, but maybe you want it recurring or doing on a specific time of the month, and you want to automate this and run this maybe every week or every month and get an output of the data that you have. So this can be really, really useful. You can add tags for permissions; we just need to specify our role, and we can create and run this job.

Now it's going to take a little bit to run because it's going to be working in a much larger data set, but in a little bit it's going to run this entire job; it's going to Output our CSV into our S3 bucket, and then we're going to go and take a look at it. So let's go over here to jobs and see; its status is still running. Let's wait for just a little bit, and it's going to be doing that on the entire uh project; it's not just going to be doing that on the sample; it's using using the uh full data set. And so let's wait for this to be done, and then let's go look at our output. As you can see, it succeeded; it took about 2 minutes; it finished up very quickly, or just a second ago. Let's come up here to our S3 bucket, and it should be right in here. So let's go ahead and refresh this. Now you can see we have this new folder, name aggregator. Let's click in it, and now we have all of these CSVs. Now this is to be expected because this is the default option within Glue Data Brew. I personally do not like this; I want a single CSV output, as I'm sure most of you do. Let's come back up here to our jobs, and let's go into this name aggregator; let's come over here, and we're going to select edit job. And now we have right here, this is our uh job that we just created. Now what we need to do is get rid of this; we need to go to settings, because right now for file partitioning, we have file output options, autogenerate files, default file output setting that generates multiple files. This usually results in the fastest job runtime, which can be important uh for some use cases, especially as you scale your uh company or your business or your department; you know, these things do matter a lot. But for what we're doing, we want a single file output. We also have an option out here for our file storage where we can create a new folder for each run or replace output files for each job run. We'll keep it the create, but depending on what you're doing, uh, you may want to replace it as well. Let's go ahead and save this; then we're going to come down here and we're going to save it as well, and now we need to actually run this. So now we're going to click run job, and we're going to run the same one except as a single file output. Let's go ahead and run this. Now it's going to be running; it's going to take a little bit longer than the minute 37 seconds, but hopefully not by a lot. So let's wait just a little bit, and then we will see what it looks like in our S3 bucket.

All right, this succeeded, and somehow took less time, uh, so AWS is just messing with us right now; they're lying to us completely. Let's go back to our bucket; we're going to come back just to the bucket; let's go ahead and refresh this. There's our next one; I think this is the second one; they have different names obviously because we didn't overwrite. Let's come into here, and now we have our CSV. You can go ahead and download that, and in fact, I'm not just going to tell you to download it; I'll download it too. So let's come in here; let's go and download this. All right, let's open this up and let's take a look. So let's come over here; let's filter this real quick cuz there's a lot in here, I'm assuming. Let's come over; it looks like there's only one, only most common name per year, uh, apparently I completely misunderstood the data; this is probably what they did was they took, they told us the most common year or the most common name per year, um, and I just wasn't thinking about it. So I think we misunderstood the assignment actually. You know what, I'll take the blame; I'll take the blame; I misunderstood the assignment. But we do have 2,779 rows, uh, and so, you know, just a little example of how to use it, although I'm not an expert on that data set. So uh, I take full blame for that, but you can see the process that we took on how to clean the data, change the data, and get a correct CSV output.

Let's go back over to AWS; let's go back to our analyst bucket, and we'll stick right here for just a second. Now one thing to note, and this is something that you may, some people may have encountered when they were trying to run this job, it may say, oh, you don't have access to the ability to do that; you may have gotten error in some way. What you may need to do, and this is something that uh my friend Kassoon over at Analyst Builder, he helped me understand, is that sometimes AWS doesn't give you full functionality until you have certain Services running, like an EC2 instance. Now we're not covering EC2 instances in this series, but you may need to come in here into the EC2, and you can just go to Services, go make sure to go to EC2, and you may need to just launch an instance and just launch it. And what happened after I launched it was it sent me an email and said, hey, you know, have full functionality for different things. It's super easy; just come in here, make sure you select your uh pair; you can just say you don't want one, and then you launch an instance, and that's it. And after that, you'll hopefully within a few minutes or, you know, within the day, you'll get an email saying you have full functionality. You may need to do this; that's just something I want to make mention of because I know I encountered that when I first did it. Now I had never encountered this before because I've used it in a workplace where somebody already had all this stuff set up. Right, we, this is a fully functioning production and development environment, and so I didn't have to worry about this, but this is a completely new account, and so this is a free instance; you know, you don't have to spend money on this because you get two EC2 free tier offers, and you can use those, and it should uh be good to go. And you need to do that for what we're about to do in just a second, um, but just wanted to mention this.

So that is how you can use Glue Data Brew, uh, and of course I want you to get in here, want you to take a look at a bunch of this other stuff. You can create rules as well for invalidating your data, and you can send yourself emails uh if you know it doesn't look good. You can come in here and look at all your data sets that you have; this is the one that we were using, the data set National baby names, and of course you can always connect to new data sets. So if you want to come in here, you want to pull data out of an S3 bucket or from Redshift or, you know, other options within AWS, you can do that. So all really, really good stuff. Now let's go back; we're going to go to all services and click up here; we are going to go all the way down to Glue. Now here's what I'll say about Glue before we even click into it: Glue Data Brew is very visual; I think it's actually fairly easy to use uh compared to Glue. Glue, I think, is a little bit more complicated, um, not entirely, but it is. Um, and so let's look at Glue really quickly. Now over on this left-hand side, we have a ton of different stuff, U, and we're not going to be looking at everything in here. Here I'm going to look at kind of the more important things, but we have something like a data catalog, and this we looked at in our lesson with Amazon Athena; they have the data catalog to set up your databases and tables and your schemas, and we had crawlers. Now this is the first thing that we're going to look at within Glue because this is how you can kind of automate pulling in data, getting the data types, and pulling in that data, and so it's really useful. The next is creating ETL jobs. So those are the two things that we'll be looking at in this lesson, but there's a ton of other things that it does. So be sure to get in here and just check everything out because uh Glue does a lot of different stuff, and I'm just not covering everything, of course, because you know this would be a 10-hour uh lesson. So we have a few different things: one, you can prepare your account for AWS Glue; here's your catalog for your data sets, and then here's how you move and transform your data. And so what we're going to do is we will need to set up roles in users; we're just not going to do it right now, but we will need to set that up, and I'm actually going to show you some of the IM, uh, you know, on the back in the IAM resource how that actually looks. But let's come over here to crawlers, and we don't have any crawlers created, but we need to. Let's create a crawler, and let's create this, and we'll call this the uh Alex the analyst crawler example. Let's go ahead and click next. Now we need to specify our data source. Now just let me back up one second; you know, when you want to pull in data to be able to use it in different services, and we want to do it kind of automatically, like I mentioned, and so let's say we're pulling in data, and we want to put it into a database, or we want to pull in data, and we want to put it into Athena; these are things that we can automate, and so uh we want to specify a data source here. So let's go ahead and add a data source; we have our S3 bucket; that's what we want to do, and this is data that we used in previous lessons. So if you didn't take those lessons, be sure to go and do that. So we're going to browse this data; let's go into the Alex anist bucket; we're going to go into patient data; we can specify single files, but

Let me just tell you, it won't work. Um, and you know that's just because of how the folder system works within Glue. So we need to specify the entire patient data folder. Let's go ahead and choose this, and now we have on subsequent crawler runs, crawl subfolders. Subfolders only if you add new subfolders to your folder then based off of an event. So an event would be like something triggering, and then you uh, it runs based off of that. So we're just going to do crawl all subfolders. Let's add this data source. Let's go ahead and click next.

Now this is the part where we need to select an IAM role. We don't have one, so let's go ahead and create a new IAM role. I'm going to call this Alex Glue Crawler Role. You can call this anything. Let's go ahead and create this. It says it successfully created it. Let's go ahead and view this. So now we're in a totally different part of AWS. This is the Identity and Access Management, that's our IAM. So right here we created an IAM role. This is the Alex Glue Crawler Role, and then down here we have the permissions policies. So right here, this is going to be giving us access to, I believe, the S3 bucket. We can come in here and actually look at the policy. So the permissions are read and write on a specific uh data source, and that's going to be our bucket with the patient data. And I think we can come up here to policy versions, look right here, and so this is what it looks like inside of it. Now this is in JSON, and you can actually customize these, and I'm not going to go into all how to do that, but sometimes you need to, depending on the data, the data source, if it's encrypted, if it's not, um, what permissions you want it to do. But what it's doing is it's allowing us to get an object and put an object in this specific resource, so that's our patient data, and that star is just a wildcard to say anything in that file path.

So if we go back and let's come back to this role, we also have this AWS Glue service role. Now this one right here, we created when we created the crawler to specify the file path of the data set. This is AWS managed; this is one that AWS creates themselves, and we can look at all the things—let me scroll down—we can look at all the things that it does uh for that AWS Glue service role. Gives us access to a ton of things for EC2 instances, S3, everything within Glue, and we can come down here and take a look at the buckets, the access to S3, as well as a bunch of other stuff. And so this is what the IAM roles look like, and you can create custom policies uh if you want to, and you can give access to different things. And so again, we're not covering this entirely in this lesson because this is not a lesson on IAM roles, but I think it is interesting and worth uh worth knowing.

So we've created our role. Let's go down over here. Let's click next, and now we need to specify where we're going to be putting this. So we're going to put this data in our Healthcare data. Say this is our Healthcare data. Again, we already had created this database back in our Amazon Athena, and we have the option to uh create a table name prefix or use the maximum table threshold for the prefix. Let's just say uh crawler, there we go. And now we have this crawler schedule down here. Now you can do it on demand, or you can specify whether you wanted a specific day of the month, whether weekly, daily, monthly, whatever you need, you can create this uh scheduler in order to run this crawler. Let's go ahead and click next, and let's just review this real quick. This is everything that we chose. We're going to go ahead and create our crawler.

So now it says one crawler was successfully created. It's this one right here, Alex the analyst crawler example, and uh it has not run yet, so we've have no crawler runs, but let's go ahead and run this crawler. So now we're going to start the crawler. It's going to start up this that we just created to pull in that patient data, and once it is completed, we're going to go take a look at it in Amazon Athena like we looked at in a previous lesson. So right down here it says it's running. So once that's done running, we're going to take a look at that data. All right, it says that was completed. Now we can come back here to the data catalog connections. Let's go ahead and refresh this. You can see we have our crawler, that's our uh prefix that we use, and we have patient_P_data. This patient data and patient data 2 were data catalogs, and part of our data catalog are tables that we created in Amazon Athena. Now let's come up here. Let's just duplicate this really quickly. We're going to go back to Amazon Athena. Let's see if it's right here. So here we go. We have Athena, and now we can access that data right here. So now if we want to, I can say uh let's just preview the table. It's running, and we have our data in here. And so this is perfect. Um, if you remember, we had our patient data and patient data 2. We manually entered that data to pull that in, and that was not super fun. So when we created, we did it from an S3 uh bucket data, but with the AWS Glue crawler, we can do this automatically, and then it'll refresh our data, and we can create this uh without having to manually do this every time we get a new data source, which let me tell you, when you start getting into a production environment or a development environment, whichever one you're in, when you have to start bringing in lots of data and the data is coming in daily or weekly or monthly, you do not want to have to manually do this. It takes up so much time. It is much better to automate this with an AWS Glue crawler. So that's during that scheduling part right in here, and we come over to crawlers, we come in here, this is just on demand, but we can edit this, and we come down here, and we can schedule it down here if we want to, but you can also—now we have it on demand—so anytime we want to run it, we can also just run it on demand. Um, so it's, you know, really, really great to have and use. So that's how we use crawlers, and crawlers are amazing. Often times, if you're using this in your work, you're going to have tons of them, like 50 to 100 of them, um, running, and you'll be scheduling them, and some of them will break because a data set changed or whatever. Um, and so this is a really great thing to test out, try with different data sets, uh, and really get familiar with it because crawlers are amazing.

Let's go down here. Let's go to ETL jobs. So this is the last thing that we're going to be taking a look at in this lesson. There is, of course, more to AWS Glue, but this is kind of like the two most important things, I would say. So let's go over here to a visual ETL, and what we're going to do is we're going to build out—I don't think I have an unsaved job—we're going to build out um our very first ETL job within AWS Glue. Now what we have here, and I'll talk a little bit more about this in a little bit, we have a parent node and we have uh the child node. I think it's node if that's the right term, but we have parent and child uh different nodes that we have in here, and we need to make sure we chain them properly. Now luckily we have this ETL visual tool, and that's what we're in right now, is we'll be able to see what we're doing when we're in here. Now let's come over here and let's go back to our sources because we have to have a source to start out with. So let's go to Amazon S3, and we're going to click into this Amazon S3. So we're going to pull data from our Amazon S3 bucket. We need to specify our location, and we can either make it recursive or not recursive, and I'll explain that in just a second. Now let's come in here. We'll go into our bucket. We'll go into our patient data. Now let's specify just this first file here. We really don't need recursive because they don't have any subdirectories, so we can turn that off, or you can keep it on. We also have a data format. The our data format is not a no format. We have a CSV format, which is comma separated. Now, now what it's kind of prompting us to do down here, and I highly recommend doing it, is getting a data preview so we can see the data as we're transforming it and changing it. So I am going to come in here, and I am going to specify this role that we had created earlier, and we're going to start this session. This could take just a little bit of time, but we're going to get a data preview down right here at the bottom. And now our data preview is ready. We're looking at uh this file right here.

Next, what we're going to do is we're going to come over here to add a node. We're going to go to transform, and we're going to to join. Now we have to specify uh a few things here. One, we can name this, so it's just called a join. Next, our node parent, and this is what I was talking about before with the parents. So this parent is our first data source, which is just called Amazon S3, which we could change. We can come in here, we can call this uh patient data 1. We can call this patient data 1. Then we'll come down here and go to the join, and notice if you come in here, we only have one data source, so we need to specify a second data source. Let's come back out here. Let's come up. Let's go back to our sources, and we'll pull in our second one. I'm actually going to change uh this view really quickly. I like it going left to right, but some people like it going up to down. Um, you can just do that using this button right here, which is the direction, but I like this direction a little better. So we're going to come into this bucket. Let's call this patient data 2, and let's specify our other patient data, which is our second, which is number two, which actually says file one in it, but you know, just ignore that. Okay. Um, but we are going to do that right there. We already have our preview, and now we need to go back to our join and specify these are the two things that we are joining together. Now if we do uh an inner join, it's not going to work cuz these data sets are extremely uh familiar. In fact, if we come in here and look at the data, there aren't any patient datas to join on or patient IDs to join on. Um, they're all unique, and so what we're going to do is we should actually be doing a union here. Let's actually get rid of this. You know what, that was my mistake. Let's go to transform. Let's come down. Let's find a union, or let me search for it. So I'm going to do a union. So we're going to do a union right here. There we go. And let's go into our Union, and we're going to specify our parent nodes. We have boom and boom, that's one and two, and it's going to be working on this data preview. There we go. Let's scroll down a little bit, and now they're all in there, which is perfect. That's exactly what we want, file one and file two, which is actually file one and file two, so we are good to go. This is looking really, really good.

Let's come back here and let's take a look at what comes next. So we've uh created our sources. We had two data sources. Let me pull this over here, and I can't pull this over anymore. Um, but we have our data sources, and again, you can pull this data in from anywhere. We've done one transformation. Now we can do other transformations as well. We can even write a SQL query. We can look at uh filling in missing values if we need to. We can aggregate our data, drop duplicates. These are a lot of different things that we looked at in Glue Data Brew, um, and there's even more options uh in here as well that are kind of unique to Glue. But lastly, what we need to do is then we need to specify our target location. Where do we actually want this data to go? And so we can put it into things like an Azure SQL database. It could go into Snowflake. It can go into a SQL Server, PostgreSQL, MySQL, Redshift, or we can just put it back in an S3 bucket, which is by far uh the most simplest thing you can do. We could also take this data and put it into the Glue data catalog, and so if we want to use this in Athena, we have a much, you know, bigger process, and then we want to put it into or query that data in Athena, we can do that. So there's a lot of different places and things that we can do. Let's just come back and put it into an S3 bucket because that is going to be the simplest thing to do. So we have our process, and there at the very end, we're going to put it into a CSV file. We don't need any type of compression here, but uh we do have an option, data catalog update options. Let me zoom in while we're here. We have our data catalog update option, so we can create a table in the data catalog on subsequent runs, update the schema and add new partitions, or we can create a table and data catalog and on subsequent runs keep existing schema and add new partitions. So I'm going to click on this one right here, and we'll specify our database, that's going to be our Healthcare data, our table name, we'll call this one uh the ETL_patient_data, and then we'll come up here for our S3 target location. So we're going to come back. We'll just place it in the Alex the analyst bucket. Let's go ahead and choose this, and we have a fully functioning ETL process. We took two data sets, we unioned them together, and then we have our output, and of course, we can see every step of the way what we're doing here.

Now if we wanted to—cuz we already have this full process right—if we wanted to, what if we wanted to add in an extra part? Let's say we wanted to come in here, and we wanted to transform this union. Let's do an aggregation. So we wanted to perform an aggregation on this. So let's go to aggregation. Now this doesn't look right, right? We can't union it and then aggregate it and also send it out. Well, we can, but it doesn't really make sense for what we're doing. So what we're going to do is let's say we want to union on maybe a diagnosis and look at the average age or maybe a treatment, could be anything, but let's come down here. Let's click on this aggregate. What we want to do is we want to put it where this is its parent, and then for the S3, the new parent for this is going to be the aggregation. So this is correct because you can see we have this line flowing here. This part is correct. We will need to aggregate it in just a second, but we need to come up here, and we need to change this parent node to the Aggregate and then get rid of the union. And so now we've changed our workflow here. We've changed the ETL process. So now we need to go to aggregate. Now we're going to come over here, and we need to be able to aggregate our data. So you have to uh select the fields to group by, and we have to perform our aggregation on a specific column, but when we come in here, notice all of these are string. We have string, string, string, string, and string, and that's not correct. Uh, that's actually incorrect. What we need to do is we need to come in here, and after this union, we need to add an additional step that's going to be our change schema, and then we'll select this aggregate real quick, and we're going to put it on the change schema. There we go. So we're just adding multiple steps here, but we need to specify what our data actually is, and we can see when we make the change what'll happen. So we can keep this, or we can make this an integer. The uh name is going to be a string. The age needs to be an integer as well. We have string for both of these, and then a file needs to be an integer. And so this is looking really, really good. Everything should be working properly. We shouldn't have any mistakes. Um, often times it'll, you know, show all blanks if you're doing something wrong. So now we can come over here to our aggregation and say we want to do this based off of the diagnosis. Then we come down to here. This is the um aggregation. Which field do we want to aggregate on? It's going to be on age, and we'll just do—let's do an average, just keeping it simple. So now this is good, and we can even take a look at this right here. So this is our data that we're going to be outputting into a CSV file, and then for our data target, we've already specified everything that we need, and of course, creating the data catalog as well. So this is ready to go. Let's go ahead and uh let's go over to job details just really quick. We want to change this. We're going to say uh first ETL. There we go. We can come down here, and there's a few different things that you can change if you want. You can change the type of job that it's going to be. You can change the Glue version, the language, the worker type, a bunch of different stuff in here, but I don't recommend uh you doing any of that if I'm being honest. The other thing while we're in here, and I'm just going to mention this although I'm not going to show this to you uh because this gets quite complicated—I do cover this in the AWS and Azure course on Analyst Builder, but it gets a little bit complicated in here—but this is the code that's being generated from your visual. So this visual is actually writing uh to a Python file. Now you can get in here and change a lot of things, um, and actually it's helpful if you want certain functionalities to get in here and change these things, but we're not going to be doing that in this lesson. I just wanted to show this to you. If you want to edit the script, you won't be able to use the visual ETL anymore, cuz that's for kind of simpler visual things where you're doing it exactly how they have it kind of made for you, but if you want to go and start doing custom things, which you can do and it's pretty awesome, you're just going to have to confirm this, and then you won't be able to use the visual anymore. So just be warned, uh, that is uh that is something. So job has not been saved. Let's go ahead and save this. This is our first ETL. We successfully created this, and now we need to run this. So let's go ahead and run this. Let's go to our Run details. This process is running, and we had multiple steps of our first ETL uh job that we created. Now we're going to let it run, and then when it's done running, we'll take a look at the output. It looks like our run failed. This happens. Let's see what actually occurs. Has an error occurred when calling the Python while writing Dynamic Frame. Access denied. Um, so let's go back. Uh, let's go to visual. Let's go see which role we actually took, or maybe uh that's in a different place, cuz maybe we need to go to the IAM and give uh different uh different options here. And here we go. So now we have this AWS uh Glue service role. Let's come over here. Let's go in and actually look at this Glue crawler role. Maybe we'll give it admin privileges, uh, just for this example, that you probably wouldn't do that in real life, but let's go over to uh let's go let's go to the EC2 cuz we don't need that one. Let's go to the IAM. So let's go back to all services. It should be...

Right over here, yep, in the security identity compliance, let's click into here. Let's go to our roles, and it should be this Alex glue crawler role. Let's make sure that's the right one: Alex glue crawler role. Let's go into this Alex glue claw crawler role. It's tough to say. Um, it is possible that we didn't have the correct access, so let's go into—I guess we we are getting, um, we're getting a lesson, a little bit of—I am—we're going to attach a policy to this. We're just going to give our guy straight up admin access, so we're going to say, uh, admin. Now there's a bunch of different admins in here; there should be one just says admin full access. Maybe it's this one? Yeah, it's allow everything, so this is the one we're looking for. So we're just going to attach this policy to this person. I think it was an S3, uh, thing. I think we just didn't have access to the correct bucket or something. Um, I'm not 100% sure, but this is saved; it was attached to the role. Let's go and try to run this again. So let's go, um, there, our visual, but we can just run this again, and then we'll go to runs and hopefully this will not fail again. If it does, we'll try to work through it. Right, these things happen; this is very common. So let's go ahead and give it a go and see if this one works this time. And just like that, we figured it out; we always do. It succeeded. Um, so it's just a permission issue, and, uh, believe it or not, that is extremely extremely common. Um, so that succeeded. Let's go over here to our query editor. Let's go ahead and refresh this. We have our ETL patient data. Let's go ahead and preview this table. Now it's being separated by commas; it didn't, uh, doesn't look like it separated out exactly how we wanted, or maybe we did it wrong, uh, I'm not sure, but it's in there; it's just not in the right format, so that's something you can definitely fix.

Let's go over to our bucket. Let's go right here and refresh this. If we come down here, we have a ton of these really tiny files, like 21 bytes, uh, really really tiny. Let's go down to the bottom. This unsave—see what this is. We have these CSVs in here. Let me—let's go into this really quick, and let's download this, and let's open this up. And so this is our original data; it looks like—so this is part of the original, uh, data set. This is not part of our actual output. Let's go back. If we come down here, let's take a look at any of these. This is just part of the Run object, and so that is not a file. Let's scroll down. So we're getting this really weird output. Let's actually go back. Let's go back to our Visual. And so for the format we chose CSV, which should be fine, uh, we have compression type as none, but maybe we need a compression type, and then we're putting this in the healthcare data, and so maybe we don't want the CSV type, or maybe we need some type of compression; maybe we need to zip this up. So let's try doing this in something like a parquet file—really great file—maybe we want to put in a Snappy compression—feeling wild. Um, I'm going to keep it like this just for now. Let's try it, but it looks like we need to aggregate this properly. And so let's go back in here. Let's choose diagnosis. I guess it forgot what we were doing over here. We'll do an aggregate function; is average, and now that's working again. So let's go over here; let's, uh, try a parquet file. Let's save this, and then we are going to run this. Let's come over here to run details, and let's let that run and see what happens. It looks like this one failed. Let's come back over here. Looks like we may need some compression. Let's go back to our visual. Let's add Snappy compression, although I'm not a fan. Let's go ahead and save this, and let's run this, and let's try again. All right, this one succeeded. Let's go down here. Let's click refresh really quick. Let's go down to our patient data; actually, we have it right here. Let's go ahead and run again. Let's go down; still doing the same thing. Let's go to our S3 bucket. I should have cleaned this up first, cuz now it's just going to keep getting messy, but if we scroll down, now we have all these parquet files. Now this is still a lot better than what we had because now we can pull in these parquet files, and we can join them together, and that's just kind of a typical run with an ETL job. Um, and so this is actually a lot lot better. If we want it all in one file, unfortunately, there is not a very easy way to do this. We can come in here and go to Amazon S3; there's not really a great way or an easy way to be able to do this. And so what you need to do is, if you want it all in one file or you want a specific format, you have to script it out yourself. Um, again, it's not crazy hard to do once you've gotten in here and you kind of understand a little bit, but if you want to be able to do some of these custom things within using ETL, this is how you do it. Um, within the visual, it kind of limits you, so this would still be good; this type of ETL process, um, would still be perfectly fine because it's just going to append all of this, uh, data. If you're putting into a database or something like that, that'd be perfectly fine. And so not as clean of an output as Data Brew was when we were working with these ones up here, but we are able to do different things that Data Brew isn't able to do within Glue Studio. And so this is how you use it; this is how you create these ETL jobs. Sometimes you got to get in, and you have to customize a little bit to get the exact output that you want or looking for. Um, but I'm going to have to come in here; I'm going to clean this up, uh, because this is a mess now, but this is how you use, uh, Glue and Glue Data Brew. Once you get in here and start trying it out and messing with all the stuff, you'll understand. Right, as you start using it more, this is not uncommon; this actually happens all the time and is not a bad thing if you know how to use the data correctly. Now, like I said, there is a ton more to AWS Glue than just these two things, although those are kind of, uh, some of the bigger things that I wanted you to know how to use, but there are things like workflows and triggers and things like schemas, and these are, of course, the most popular ones, so make sure to know these ones, and we covered some of them in this lesson. But Glue is very expansive, and you're going to use it for a lot of different things. Now, if you're a data analyst and you're working on something like a data collection team, U—which is something that I worked on for many years—then you might be getting in here and using this quite a bit. But if you're just a data analyst and you're waiting for the data engineer, database developer, whoever's, you know, bringing in this data, if you're waiting for them to bring in the data, most likely you're not going to get in here that much. You might inform them; you might say, hey, I was looking at the data, uh, the data looks bad, or there was something wrong with it. So you're looking for quality issues immediately; then you'd lay that to a data engineer and say, hey, could you go and check out that job that brought in that data, and then they would go and check it. That's typically how, you know, that would actually work. So just something to be aware of, uh, with Glue, but, uh, Glue Glue Data Brew and Glue crawlers, as well as ETL—all stuff that I recommend you testing out, trying out, figuring out how it all fits together and works. With all that being said, that is the end of the lesson. If you haven't already, be sure to check out my AWS and Azure course on Anal Builder. I go a lot more more in-depth into Glue and everything in it; we even create some custom scripts in here to put it all in one file, which is really, uh, useful to know how to do. So if you're curious about how to do that, be sure to check out that course. If you have not, be sure to like and subscribe below, and I will see you in the next [Music] video.

What's going on, everybody? Welcome back back to another video. Today we're going to be taking a look at QuickSight in [Music] AWS. Now QuickSight is AWS's data visualization tool, and so a lot of companies, when they get into AWS's ecosystem, they have really simple needs for their data visualization, and they just use the internal tool, which is QuickSight. It's not the most robust tool you've ever seen in your life, but we're going to take a look and see how it actually works, and I'll make a lot of comparisons to things like Power BI and Tableau, which are really popular data visualization tools. So without further ado, let's jump on my screen and take a look. Let's scroll down. Let's go to analytics, and under the analytics we have QuickSight right here. Now, in order to get started with QuickSight, we actually have to create an account, so we have to sign up for QuickSight. So let's go ahead and say sign up for QuickSight. For the authentication method, we just want to use the IAM Federated identities. You can, uh, use some of these other ones if you'd like. Be sure to specify your region; make sure you fill out your account info, and then down here you can, uh, use an existing role, or you can just use a QuickSight managed role. Highly recommend just using default; much easier, and allow access and autodiscovery for these resources. So if you want them to be able to use your S3 buckets, be sure to click on your S3 bucket, and we can say, uh, here's the bucket we want, uh, you know, maybe I'll just choose both of them, but here are the buckets that we want you'd be able to access, as well as all these other things as well, and then we'll come down here, and we don't want paginated reports, and then we'll click finish. So now it's creating our account, and then we'll get access in just a second. All right, that took about 10 seconds. This is the name that I gave, uh, our account for QuickSight, and let's go ahead and go to QuickSight. This is what's new in QuickSight. Let's go ahead and close out of that. Now this is the UI for QuickSight. Now, just before we get into, you know, creating stuff and building stuff and all these different things, I just want to talk about QuickSight in general. This is AWS's tool for a visualization, and of course there are other options; there's Power BI, there's Looker, there's, uh, Tableau; there's lots of different options, and all those are perfectly acceptable, and in fact, in some instances they may be better for some things, but within AWS's ecosystem, this is their BI data visualization tool, and so because it's already integrated into all their services, it does make it a little bit easier to use than some of the other ones, um, and that's what they're hoping for; they're hoping you use their services and stay in their ecosystem, so you don't have to anywhere else for any service ever, because AWS wants your money and your business, of course. Now, within QuickSight, you can share reports; you can create folders that you can, uh, share with customers and clients; you can create dashboards; you can, of course, create stories. We have analysis, data sets, and topics, and then over here we have some sample analysis. So these are different, um, things that they've already created visualizations for, but the first thing I want to show you is right over here in data sets. Now these are the data data sets that we already have access to: sales pipeline, uh, let's just click in on one. We get a very slight overview of kind of what this data looks like. We can refresh this data, and so if we're connected to a data source, we can refresh this, or we can schedule this, and then we can look at permissions and usage for people who have access, uh, to this data and how they can use it as well. So just some interesting things within data sets. If we want a new data set, we just come in here, and we specify where we want to pull in our data, whether it's an S3 bucket or we want to upload a file or it's coming in from Athena; there's a lot of different places, uh, that we can pull it in. There is something up here I just want to note: we're not actually going to be pulling in any data for this; we're going to be using one of their samples just for demonstration purposes, but there's something called SPICE in here. SPICE actually stands for super fast parallel in-memory calculation engine, and it's used to do things really quickly in QuickSight. But for example, let's just say we were, uh, uploading a file; we can choose use, uh, SQL database output. I don't even know what's in that file. Let's see what the preview says. Okay, we have some products; this is our products file. Let's say we select next; we want to bring in, uh, this file; it's going to import it into SPICE, and so that's going to cost money; that's going to put it into there super fast parallel in-memory calculation engine, and then you were going to pay for that service. Now that isn't always the case; we don't have to, uh, we're not going to do that; we're not—we're going to cancel that import, but you don't always have to do that, especially you have other types of data connections to like MySQL or PostgreSQL; you may not need that, but just be aware of what that is, cuz that does cost money. Let's go up here to QuickSight. We're going to come on back, and let's take a look at some of these samples that they have. Let's go into this sales pipeline analysis. So right now we're working with sample data, and we're not going to be doing a full project in here in QuickSight. Really what we're doing is just seeing how it works and how everything is built out and how you can customize and build out your own dashboards, analysis as well. So if you come in here, you can see that we have different types of visualizations—very standard types of visualizations, uh, for any BI tool—and if we click into one of these, let's say we're clicking into this one, you'll see we have an X axis, a value, and a color on the left-hand side. We also have all of our fields, and you can see whether it's a dimension or if it is a measure. Next, you can see that up here we have our data set; this is the data that we are using, and right over here is the type of visualization chart. Now this should look really familiar if you're familiar with Tableau or Power BI, both of which I have series on YouTube. These are different types of visualizations that you probably have seen, and so if we wanted to, uh, change this, we can click on a different type of chart, and it's going to visualize that data based off of that visual type. It's that going to be extremely extremely useful. These are ones that I would use all the time is just clicking on one of these and then filling in the x-axis, the value, the color, and all of these different things that it's prompting you to provide. Another thing you'll notice is we have different sheets up here, so within this analysis we can add new sheets, so we'll make this, uh, interactive, and now we have a blank sheet here that we can build our visualizations off of. So let's say we wanted some type of donut chart in here; we have our donut chart, and it's going to, uh, prompt us to fill in what it actually needs, so we need a value, then we need to group SLC color. So let's say we want the, uh, region to be group and color, and for the value we'll do the weighted, uh, Revenue right here, and we're going to do the sum; that should work, and now you can see that this is interactive; you can go ahead and click on this; you can click into these different options; you can change these colors. Let's say we want to make this one, uh, black. There we go. And there's other options up here like format this, uh, visualization, or if we want to maximize it or minimize it, and if we want to add any more visualizations, we just click add, and we can add it right over here. And so everything in here is quite customizable, and you can rename these sheets. Let's call this one, uh, sum by region; call it whatever we want. This is just for this one, uh, visualization. Everything in here is quite customizable. Now, once we have our visualizations, we have this properties right over here, and we can change a lot of the titles, subtitles; if it's a donut chart, we can make it large, small, medium; we can show the total versus not showing the total, so everything in here is very very customizable, and we can specify if we want the data labels and the legend on. So if we don't want the legend, we can turn it off and on, or if we don't want the data labels, we can turn those off and on as well. We also have over here an interactions tab, so if you want to customize some of these interactions or if you want to customize some of these tool tips, you can also do that as well, which happens when you hover over it. So right here on this right-hand side, it's giving us the, uh, field for region and the weighted, and you'll notice when we hover over it, you can see it's the US with the weighted Revenue. So we can specify what we want in here; maybe we want to, um, add something else; maybe we want to add the date—doesn't make sense for here—but maybe the, uh, segment. Let's go ahead and do that, and we hover over it; you can see there's a count of the segment, which is 12824 for this data set. So you can customize these tool tips as well. Now, when we're done with our visualizations, we can come over here to file, and we can publish this and save as an analysis. So just saving is going to save it for you to pick up later and do whatever you want with it, but let's say we want to publish this. Publishing this dashboard, we're just going to call this, uh, sample dashboard, is going to make it available to everyone. So let's go ahead and publish this dashboard. So now we have, uh, this first sheet that was created by them, but then we have the second one that was created by us. Now, once we've published it, we can then come over here, and we can share this. So we can share this dashboard, and we can come in here, and we can choose anyone who has access to QuickSight; we can say I want to share this with my colleague or the customer, whoever it is, and I want to use this. The other thing that we can do is we can copy this embed code. What you can do is you can say, hey, I want to embed this on my website or on this platform, and I want to, uh, show this visualization; you can embed that using the code, uh, from this right here. We also can copy this link, and so we'll copy the link; we come right over here, and now we have access to this dashboard. So anybody who has access to it can just use it as a link, and that's really helpful as well. But what I thought was really interesting, uh, within all this is the embedding code, because not every BI tool is embeddable, but QuickSight, because of course it's with AWS, it's all internet-based; it's not, you know, local to your computer; it's very embeddable into almost, you know, anything you want to put it into. So let's go back to our sample dashboard; we'll go back to QuickSight. So that's our different analysis, but let's go over to our dashboards. When we actually published our analysis, it went right in here to our dashboards. Now this a completed dashboard; we would again share with our internal teams or our customers. We also can create data stories. This is something you do need to upgrade in order to receive access to this, but this is very very similar to data.

Stories in something like Tableau. If you've watched my Tableau series, it's very, very similar. You can watch this uh video as well, but you create these little stories. You can add narratives and you know why the data is doing what it's doing and what impact that has, and it's pretty good. I personally don't use data stories all that much. I mostly would create a lot of KPI metric dashboards and stuff like that, but you know there are use cases for data stories, of course.

Lastly, we have these folders. Let's go ahead and create a new folder. Let's call this one "Alex the analyst" folder. Once that folder is created, we can come back to our dashboard. We can right-click on this and say "add to folder". Then we a specifier our folder, so we'll go to my folders and we'll add this. And so now within my folder, if I click in here, I have this dashboard. Now this is really just an organizational tool for the most part, but when you start using this in a production environment, you're going to have a ton of different dashboards. And so you can come in here and create different ones for different customers, different clients, internal teams, so that you can organize all these different dashboards and data stories and whatnot, so you have it all in one place. And so again, that's just super useful, really user-friendly.

Now this is really all we're going to cover in this video. We're not doing a full project in QuickSight. I'm really just demonstrating how to use it, some of the things that they have in here that I think are really interesting and how you can publish and share and do all those things within QuickSight. So I hope that that was helpful, and if you have not already, be sure to check out my full Azure and AWS course on analystbuilder.com. And if you like this video, be sure to like and subscribe below, and I will see you in the next video. [Music]

What's going on, everybody? My name is Alex Freeberg, and today we're going to be walking through my top three tips on how to use LinkedIn to land a job. LinkedIn is a fantastic place to look for a job; it's its own little ecosystem where career-driven people can connect and talk with one another and help each other find jobs. I personally have landed jobs through LinkedIn, and so I know how effective it can be. Let's jump over to my screen, and I'm going to show you my top three strategies that I have found to be the most successful to actually finding a job.

So I'm logged into my completely anonymous account here, and I'm going to show you the very first tip, which is you shouldn't be just applying to a position; you should be actually reaching out to the recruiter. And I'm going to show you exactly how to do that. So the first thing that we have to do is actually find a job that we want to apply to. So let's go to the job section right over here and let's search for "data analyst," and let's do that in uh, let's do Chicago, cuz why not? Uh, so it's going to search for data analyst positions in Chicago. We have one right here. Let's see what it looks like, cuz you know I don't want to apply to jobs that I'm not extremely qualified for. So this is a job that I want to apply for, and before I actually go and apply to the job, I want to see if I can reach out to a recruiter and talk to them beforehand. So let me show you how to do that.

So what we're going to do is actually click on the company right here. It's going to take us to basically their LinkedIn profile page for their entire company, and we're going to scroll down; we're going to go over to "people," and then we're going to search for "recruiter." So if we scroll down all the way to the bottom, we can see that there are recruiters that actually work in-house for this company. And so now would be a time where I actually reach out to some of these recruiters and I say, "Hey, I see a job that I really like; I think I'm really qualified for, and I would love to talk more about it with you." You can ask them things about the job to make sure that it is a good fit for you, and then I highly recommend you ask, asking them what they think is the best way to apply for this job to make sure that your resume gets noticed and you get an interview. Since they are a recruiter who works at this company, they may be the one who's actually going to be looking at these résumés, and so they may give you a tip on the best way to actually apply. They may also just ask you to send them your resume directly so that they can look at it, or maybe later on down the line this actually is a person who is reviewing résumés, and so if they come across your resume, they may be able to put a face to the name, and that may give you bonus points. I'm going to leave a template script in the description in case you don't know exactly what you want to say to this recruiter, and it'll give you just a baseline of some of the things that you might want to say.

Number two is to actually ask for a referral. Now, if you don't know what a referral is, it is where somebody who already works at the company can refer you to a specific job, and that might get you a little bit higher on the list for interviews. So I highly recommend reaching out to somebody who already works at that company and ask if they're willing to be a referral for you. I get people reaching out to me all the time asking to be a referral for them for my company, and nine times out of 10 I say yes. I always ask to see their resume first, just to make sure that their resume aligns with the position at least a little bit, but there's basically no harm in me being a referral for somebody. In fact, I may actually get a bonus if that person ends up getting hired, and so for the most part there's almost no risk for the employee to actually being a referral, and so a lot of times they will say yes. Now let me show you how to do that, and it is very similar to finding a recruiter. So we're going to stay on this "people" section, but instead of searching for a recruiter, we're going to search for a job title that is similar to yours. So let's actually see if they do already have any data analysts, and if they do, that is the person that we're going to reach out to, because that is the person we'll probably have the best connection with. So it looks like we have six employees, and let's scroll down, and so it looks like all these people have data-related jobs, and so I would reach out to these people and say, "I saw an open data analyst position at your company; I would love to know more about your company as a whole," and then you can talk to them a little bit, and then in the end your goal is to ask them for a referral, and if that happens, that is fantastic, and then you can go ahead and apply for the job and mark them as a referral for you.

Now my third tip on how to get a job through LinkedIn is to actually have recruiters reach out to you. So let me show you how to do that. The first thing we're going to do is actually go over to my profile here, and we'll click "view profile." Now there's a few things that we want to make sure that we have on here so that recruiters can reach out to us. The first thing that I want to do is to actually come to this section right here, which is "show recruiters you're open to work," and when I click on this, I can actually choose some job titles and some locations where I actually want to apply and have recruiters reach out to me. And so right now I have data analyst; I have in the DFW area, which is where I live. I can also add titles like business analyst, um, and then maybe junior data analyst, entry-level data analyst, or things like that that could potentially have recruiters reach out to me for positions that I'm interested in, and then you can say that you're immediately and actively applying, and you can also say that you're only looking for full-time positions or contract positions, and then you can actually add this to your profile, and I only want recruiters to see that because I do currently have a job at McDonald's, and so I don't want McDonald's firing me because I'm looking for employment elsewhere. So let's save that, and it looks like it was updated. And so now when recruiters are searching for candidates for a specific position, you will be on that list so that they can find you and reach out to you.

Something else I should mention is on your profile page, I would try to have some type of professional photo so that you look really good. I would also try to include "data analyst" somewhere in your title. If you already have a data analyst job and you're looking for another one, you can just have your previous company, but if you're looking for a data analyst job, you, you can always put "seeking data analyst position" or something like that. Another thing I think is really important is having really good descriptions for your previous work. I don't currently have this, but I would go a little bit into the work that I actually do; make sure that the experience matches kind of what you're looking for, if you do have previous experience; if not, that's totally fine. The next section on your profile page that I would recommend looking at and updating is your skill section, and so you want to go in there and make sure that you have all of your relevant, really data analyst-heavy skills on there, specifically hard skills, because soft skills aren't going to translate too much into this section. I would definitely stick to things like SQL, Python, Tableau, Excel, things that data analysts are going to use, because this is where they're going to actually look and see if you have the skills that they are looking for for that position. When I was applying to jobs and only applying to job postings and not using any of these strategies, my success rate was 0.04, which means out of 1,000 applications that I filled out and sent my resume to, I only heard back from four of them to actually get an interview. But with these strategies, I was able to get that up to 10%, and at my best I was able to get that up to 15%, but that's because I was applying to a lot less positions, and I was targeting jobs that I really wanted to work for, and so I put in more effort in order to contact people and work with recruiters in order to get that job. I genuinely hope that these strategies can be helpful for you, especially if you're trying to apply for jobs right now. Thank you guys so much for watching; I really appreciate it. If you like this video and got anything out of it at all, be sure to like and subscribe below, and I'll see you in the next video.

Hello everybody. Congratulations! If you are watching this, that means that you completed the data analyst boot camp. If you haven't, don't keep watching; this is only for people who have completed the data analyst boot camp playlist on my YouTube channel. Woo! All right. Now that we filtered those people out, I'm going to show you how you can download your certificate and your certification. Now that you've completed the data analyst boot camp, I will leave a link in the description, but let's go on to my screen. I'm going to show you how to actually access this and download your certification.

All right guys, don't go around telling people this or sharing this, uh, but this is our data analytics boot camp on the Alex the analyst GitHub right up here. I will have this link in the description. What you can go ahead and do is you can come right here, and you can download this; you'll just right-click or click "download" and just do something like "save image as," um, or you can come to this one; this is the one that I think is the, the real money maker here; this is the certificate of completion for the data analytics boot camp. I have my, not signature, but my name, as well as my position, with a blank space right here to fill in your name. Feel free to put this on LinkedIn or Twitter or Instagram and tag me in that because I would love to just say congratulations, because honestly it's a lot of work to go through all those videos and learn all of those skills. So congratulations! I hope that you learned something along this journey, a new skill, a new thought, a new idea, and I'm proud, proud of you. I'm proud of you for putting in the work; it's not easy, but you did it, and I hope that you came out on the other side better for it. So congrats! I'll see you in the next video. [Music]

Get Our Mobile App

Ultimate Data Analyst Bootcamp [24 Hours!] for FREE | SQL, Excel, Tableau, Power BI, Python, Azure

Transcription