Azure Databricks End-To-End Project 2025 [From SCRATCH!]

This is the only Azure Databricks end-to-end project you need in 2025 to crack the interviews and master Azure Databricks technology. Really.

But why should we trust you? Because this project will tell you how to incrementally load the data using Spark Streaming and ingest pipelines. Then, it will demonstrate all the PySpark functions, including Python OOP concepts such as classes, and using Unity Catalog functions to populate the silver layer. And in the code layer, you will build a star schema and will build slowly changing dimensions, including type one and type two, both.

And not only this, you will also learn about Delta Live Tables, which is the most in-demand technology right now in the world of Databricks. And wait, there is one more thing. You will build end-to-end ETL pipelines and workflows in Databricks. I think now you can trust me.

But why I covered so much in this particular project? Because Databricks is one of the most in-demand technologies right now, and I want you to master it. And I said master it because it's very easy to get started with Databricks, and it needs serious attention and focus and strong hands-on to master Databricks. And I have taken the responsibility to make you a pro developer in the world of Databricks.

So if you want to support my dedication, you can just hit that subscribe button and just like and comment on this video. And just take out your notebook, prepare your coffee, and let's get started with this end-to-end Azure Databricks project.

So what's up, what's up, what's up my lovely data fam? What's up? And what's up strangers? So just hit that subscribe button, then I will also welcome you as, like, welcome my data fam. What's up? Okay, so just hit that subscribe button because, because, because, because you're going to love this video, trust me.

Why I am saying this in the beginning of this video? Because I'm recording this introduction part at the end. Because I wanted to just build the entire project with the flow, and I wanted to keep it raw. And let me just give you a little spoiler. This video is amazing, bro, amazing. Like the things that we have covered, let me just tell you.

Hold on, hold on. I know you are also excited, that's why you just clicked on this video. And second thing, I know you are really, really dedicated in order to achieve your dream job, dream role, and obviously, you want to be placed in your dream company. That's why you just clicked on this video. So by the way, you just need to thank God because obviously they just made you click on this video.

Okay, so let me just give you the whole architecture. What we're going to do? Because I cannot wait. Because I have just completed this video. I have just completed the recording. And bro, this video is insane. Okay, let me just tell you what we're going to do. So we are mastering Databricks in this video by doing, or by you can say, creating or building an end-to-end project. When I say end-to-end project, I mean it. Plus, we want to create a real-time project, real-world project. And let me just tell you one thing, bro. There's a difference between a hobby project and a real-world or real-time project. This is the real, real-time project, trust me.

Why? Because of so many reasons. You will get to know along with this video. Because when you just build a hobby project, you do not perform administrative tasks. You do not perform security parameters. You do not allow anything, you can say, cross-data communication between the applications. You do not do all those things. But in this project, you're going to do everything.

So let me just tell you about the architecture. What we're going to do? So first of all, we're going to obviously master this particular technology, which is Databricks. Okay? In this technology, we're going to build a medallion architecture, which means our data will be distributed in bronze, silver, and gold layers. Okay? And our data will be coming from GitHub. Okay? And Azure. Yes.

And let me just give you another spoiler. We're going to perform incremental loading, which will include item potency, which means exactly once criteria. That means once you process your data, you do not need to perform your data, like, process your data again. We're going to perform Spark Structured Streaming in order to just perform incremental loading. So for your bronze layer, you're going to learn incremental loading, Spark Structured Streaming, item potency, and the fourth thing is automated ingestion pipelines, which is a new feature in Databricks. So everything you will learn in this particular layer, which is bronze layer. And the best part, we're going to work with columnar file formats, which is also called as big data file format, instead of just CSVs. So we're going to work directly with Parquet file formats, just for your information.

So the moment you have dumped your data in your data lake. Okay. Okay. Makes sense. How you're going to do this? We're going to use something called as Unity Catalog, which is the most in-demand architect, not architecture, you can say a kind of data governance solution in the modern world. And trust me, Unity Catalog is the backbone of Databricks right now, and backbone of all the Databricks interviews as well. Just mark my words. Okay? Just mark my words.

So we're going to leverage Unity Catalog. And plus, as I've just mentioned, you're going to perform all the administrative tasks as well. Because nowadays, companies are looking for those developers who know a lot of stuff, like stuff, not staff, lot of stuff. Who can just perform administrative tasks? Who can just perform, obviously, development tasks? Obviously, in the organization, they have dedicated people, but they want talented people who will be knowledgeable enough to perform those tasks in the absence of those people. That happens. Okay? So you need to have that knowledge as well.

Then let's talk about our silver layer. This time, this silver layer is really, really, really helpful and really important. Why? Reason number one, we're going to learn so many in-demand PySpark functions. Okay? Reason number two, we're going to learn how we can leverage OOP concepts in Python, and we're going to integrate those concepts with our PySpark code. Wow, amazing. Third reason, we're going to learn something called as functions in Unity Catalog for reusability of code. This is again, a new feature, um, in Databricks. So you're going to master that as well. So there are three reasons that make this silver layer really, really, really special. And trust me, this silver layer is really, really, really special. And we're going to save our data in the Delta format in this silver layer. Okay? Makes sense.

Then, once we have our data in the silver layer, then the main part, I would say, all the parts are main, but this is the most important part of the video, which is the gold layer. Why this is the most important one? Because you can already see what is written on the screen. We're going to build a star schema. That means we're going to create dimension tables. We're going to create fact tables. We're going to build slowly changing dimension type one. We're going to build slowly changing dimension type two. And we also going to learn Delta Live Tables as well.

So slowly changing dimension type one, you will be building by code, by writing each, each, each line of code. And slowly changing dimension type two will be created by Delta Live Tables. And you will actually have the end-to-end understanding of the gold layer. And this is again, the most important and the most, most, most in-demand area currently in the world of Databricks, which is Delta Live Tables. So you will master Delta Live Tables as well. Oh, really? An amazing video, bro.

Then, obviously, once we have our data, um, sorry, star schema ready, then we'll be just using warehouse. Yes, we have warehousing capabilities available in Databricks as well. And obviously, there are some outsider tools such as Synapse, Snowflake, and so many other tools as well. And we're going to share those endpoints with the Power BI. And obviously, we do not need to build the Power BI dashboard from scratch. No. But yes, being a data engineer, you should know how you can just establish the connection with the Power BI dashboard so that your data analyst can efficiently build the reports on top of those dashboard sources. Huh?

So this is a very high-level overview. Just to, this is not even 1% of the video. Just trust me, because we have covered so many interview questions in this. Because I was just creating the video, and I just covered, "Hey, this is your interview question." "Hey, this can be an interview question." So there are so many tips, there are so many tricks, there are so many debugging strategies as well. Like this video is full of, full of, full of knowledge. And trust me, once you build this project and you completely understand what I am doing in this project, this project will actually make you an outlier in the interviews. And this project will actually help you land a job. Trust me. You just need to explain this project with confidence in the interviews. And that's it. Because we have followed all the real-world strategies and approaches. And you can easily say that you are well-versed with all these areas. And trust me, you can answer all the questions because we have covered so many things in this video.

So this is just, just, just one person, you can say, trailer of our video. Okay? So now you will be saying, "Anlama, we are fully charged up. We also want to build this project." So what are the prerequisites? So basically, prerequisites here are the prerequisites. So first of all, you should have a laptop or PC with a stable internet connection. Okay? Stable or like, almost stable. Okay? So then, second thing is Databricks account. If you do not have any Databricks account, don't worry. We'll be creating Databricks account for free. Plus, we're going to attach a cloud storage as well with that. Because obviously, in the real world, we attach any cloud solution with Databricks. Okay? So we're going to just create both the accounts for free. Don't need to worry.

The third thing is very, very, very important, and this is non-negotiable, which is excitement to learn Databricks end-to-end solution. Because Databricks is very easy to start, but it is not very easy to master. You need patience, you need excitement. And I'm not lying. Like, uh, we do not need excitement. We can just simply learn it. Just try, try, learn, try to learn it, and just see the results. Without excitement, you cannot learn. Forget about Databricks, you cannot learn any technology. Trust me, trust me. So you should have the excitement that you are learning something new. You are learning a technology which is very much in demand. You should feel excited because once you master this technology, you will feel like, "Okay, now I know a lot of things in this particular area which is in demand, which no one, like, not everyone knows about." So you need excitement for it, right? You need excitement for it.

So now, I think that's enough introduction. Now let me just give you a quick overview of the data that we're going to use, all the repositories that we're going to use. And now let's see what do we have. Because I know you are all set and just hit that subscribe button right now and share this video with others and do like this video and obviously do comment on this video. This will help me a lot to grow. And if you want to support my dedication, I'm literally, literally, literally putting a lot of efforts and time to create these videos so that you do not need to pay for those, like, thousands of bucks to anyone. Just support my dedication. Just share this channel, comment on the videos because it helps me to grow. Just like the like the videos. Just do everything that you can and just support, support me. That's all I can say.

And let me just show you that data right now. So basically, this is your repository from where you can just download your data. So this time, we are not going to work with CSV files. Why? Because you are becoming a data engineer in today's era, or you want to just excel as a data engineer in today's era. So obviously, CSV files are obviously important, but those are just text files. So you should know like how to work with the columnar-based file formats, which is like Parquet file format. So you should know like what are those big data file formats and how we can just work with Parquet file formats. So what I did, I just took the CSV files and I just converted them into Parquet file formats. Plus, I just divided that data in the incremental way so that you can just feel like how we can just incrementally load the data. So we are just building incremental flows, incremental processing, incremental notebooks, everything. And at the same time, you will actually validate the data as well because we have incremental data files in the form of Parquet.

And just to give you an overall glimpse, like what do we have in this particular files. So basically, we are going to work with a retail dataset in which we have something called as orders, in which we something called as have something called as customers, we have something called as products. Why did I just take this dataset? So basically, you are going to master Databricks today. And when you will you, when you will be just aligned with the dataset, you will learn any technology with a faster rate. And trust me. And obviously, now you are becoming a data engineer, or let's say you are already a data engineer, but you are mastering Databricks, you would have already worked with these similar datasets, maybe in your school days, maybe in your college days, maybe in your current job, like orders, customers, products. You can actually feel when we just join those tables, you will actually feel what is there in the dataset. Let's say I will just pick any dataset from IoT dataset, you will say, "Hey bro, what is this IoT dataset? We do not know actually anything." So that is why when you are learning, it is very important to pick an easy dataset because you are not mastering any kind of data sources. No, you are mastering the technologies. You are mastering the processing or end-to-end flow. So you should just pick those datasets with which you can just feel aligned. Okay? Do not pick complex datasets. Do not just pick those datasets which will make you feel happy. Okay, this is my order. Okay, this is the product. Okay, this is the category. Okay, this is the customer. You will feel aligned. Okay? So your learning graph will definitely go up with this approach. So that is why I just picked these datasets.

And as I just mentioned that you'll be just working directly with the Parquet file formats because in the real world, you will rarely work with the CSV files. You will work with CSV files, but Parquet file formats are something that you should know how they work under the hood and how you can just actually process the Parquet files. And trust me, once you are a data engineer, you're going to hate CSV files and you're going to learn Parquet files. Why? Because those text files do not hold any kind of schema, but these Parquet files actually store the schema at the footer of the file. See, this can be your interview question. So we're going to just cover these little, little interview questions as well along with the video. So do not need to worry. And obviously, these information are really, really helpful when you are just sitting in the interviews. Person can just ask you a simple question that will, that will not be directly aligned with the, you can say, role, but yeah, these information, or you can say, these pieces of information are something that you should know, like you must know. And obviously, these are not written anywhere in any kind of roadmap. But yeah, you should know all these things. So that is why I'll be just mentioning some of the little, little questions that you should know along with this video. And you, you already know about me, bro.

So this is the dataset that we are going to work with today. And you can simply download all the files from here. Simply go here. And obviously, you will not see any kind of preview version because this is a Parquet file format. Simply click on this download. And your file will be downloaded. Just click on it. And you will see, "Hey, it is downloaded." I have already downloaded it. So it's fine. And that's it for the data.

And once you have all the data ready in your PC or in your local computer or any laptop, then now it's time to create an Azure free account. You will say, "Hey, why we are just going to create an Azure account?" Because we are going to work with Databricks, right? Yeah, exactly. But in order to work with Databricks, you need to have a cloud account associated to it so that you can actually see how things are going to work. Because when you are going to master Databricks, you need to just attach that instance with any kind of cloud. And Azure is the most in-demand cloud right now. And let me just tell you one thing. When you just attach any cloud with any kind of, you can say, processing, uh, solution, and we are obviously learning Databricks, so in that particular scenario, you need to work with so many administrative tasks such as like, um, allowing Databricks to read the data, creating external data locations, creating Unity Catalog, creating, blah, blah, blah, so many things, bro. And these are the things that create the difference between a normal hobby project and a real-time project.

So I know it's very easy to say, "Hey, this is a real-time project." There's a definition of real-time project in which you actually build everything from scratch that you build in the real world. That is called real-world/real-time project. Okay? So if anyone, like, let's say, um, tells you, "Hey, this is the real-time project. We are just building a real-time project." There's a definition. What is the meaning of real-time project? So you need to configure all those things, like security parameters, your, um, permission configurations, your Unity Catalog, your catalogs, your schemas, your tables, your medallion architecture, everything needs to be done by you. That is called real-time project. It's not just like opening Databricks, writing few lines of code. It is like real-time project. No, bro. Real-time project is something else. Okay? And this is the real-time project. You will get to know why? Because we're going to cover almost everything in Databricks. And you got to master Databricks today with this project. And we will be using Azure Data Lake as a storage solution. And just as a kind of data lake solution in order to just preserve our data. Makes sense. Okay, very good.

Now, let's create our first. If it is your first Azure account, if not, obviously you can just use your existing Azure account. And if you do not have any Azure account, let's create the first Azure account. And now let's get started with our project, bro. Let's do it.

So now, in order to create your free Azure account, I hope that you would already have one free trial. But this is just for those who do not have a free Azure account or who are just coming from a different cloud environment or different cloud background. So do not need to worry, bro. I am here. So how you can just create? Simply go to your, um, Google tab or any tab. Simply search Azure free account. Just search. You need to type "n" as well. Okay? So simply click on the first link. And then simply click on this "Try Azure for free." That's it. And here you will be landing on the page where it will ask you to just provide your email ID and some credentials. So basically, if you have a Microsoft account, okay, then simply put it here. If you do not have, simply click on "Create one." And you just need to register your normal Gmail account under Microsoft. And you can even use @gmail.com or @outlook.com. So if you want to just create a new Outlook account, you can even create that one as well. I don't know, I don't know, I don't know. Just get your account and that's it. Okay? Simple. Okay, very good.

So now, just keep your email ID with you because obviously you need to put it here. Then when you'll be clicking on next, it will take you to a page where it will ask you to complete a kind of web form. And it will just ask you your personal details such as name, um, phone number, email ID, and blah, blah, blah. And at the end, you will simply click on "Sign up." And the moment you will click on the sign up, it will just ask you to just provide some card details and some billing instructions. Do not worry. They will not send you a bill because this is a free account. But yes, they need to just verify that you will be the one who will be using the services. Because the thing is, Azure gives you $200 USD credits in order to use their services. So they need to just verify, "Okay, this is the right person to whom we are just giving the credits." And you are the authorized person to use these credits. And these credits will be there for one month. And after one month, it will be gone. So your question will be like, "Hey, what will happen after 30 days?" So nothing will happen. They will simply nullify your services. And and you do not need to worry about anything. They will not send you any bill.

So this was just a prerequisite that you need to create a free account. Okay? So once you create a free account, let me just take you to the Azure portal account. And simply search portal.azure.com. This is the link that we use. Hit enter. Then I will simply select my email ID. And I can easily log in to my portal account. So now you need to put the same email ID that you have created just now. Okay? Got it. Simple.

So let me just take you to the Azure account. And let me just give you a quick overview of it. Let me just show you. So this is my Azure portal account. And this is the homepage that you will also see in your particular account. And don't worry, in your case, it might be a little different because of these Azure services that you will not be seeing here. Maybe you will see some different services. So you do not need to worry. These are some of the recent services that you see if you have used in the past. And obviously, if this is your new account, that doesn't make any sense if you will see these services, right? Common sense, common sense, common sense.

So now, I know this UI is so, so, so amazing. I personally love this UI. And now let me just give you a quick overview of this Azure. Do not worry, we are not discussing Azure in detail. We are only focusing on Databricks, data, Databricks. But yeah, you should have some understanding. Basically, uh, this is the area, like maybe you can call it as like "Recents" page. Because I know this is a homepage, but this is like showing all the recent tabs that we open. And let me just take you to this menu bar. Basically, these are some of the resources which are really popular. And we use on a daily basis. That's why they have just put it here, like these resources. And the one thing that you should know, even if you are learning only and only Databricks, this one thing you should know. What's that? Basically, basically, basically, basically, all resources and resource groups. What, what are those? These two things. Basically, resource group is the backbone of your resource management. Why? Because the thing is, anything you create, or let's say, anything you use as a service from Azure, it's called a resource. If you're using Azure Databricks, it is a resource. If you're using Data Lake Storage account, it is a resource. If you're using Azure Data Factory, it is a resource. So every resource, every resource needs to be added in a folder. And that folder is called as a resource group. Simple. Just keep it simple. Do not make it complicated. Life is already very complicated. So do not make it complicated by saying, "Resource group is like hard construction in which we just store our resources." Shut up. So just, this is just a folder. Same folder that you create in your machine to just save the files. Similarly, this is a folder in which you just put your all the resources. That's it. That's it. That's it.

So in order to create any resource, because obviously in our case, you'll be creating, uh, Databricks, obviously you'll be creating Data Lake, so all those resources need to be added in a folder. So let's create that resource group. And how you can just create that? Simply click on this search bar. And simply search "resource group." And as you can see, "Resource groups." And I have so many resource groups because I'm lazy enough to not, not to delete these when I just create these. So just click on "plus new." And let me just show you how did I clicked on this. See, "plus create." Click on this. And then click on "Resource group name." So obviously, you need to provide a name. I will simply say "RG" and then "Databricks E2E Project." Or let's say "ET End to End." Makes sense. Catchy name, right? So simply click on "Region." Um, region, let's pick "UK South." Why? Who lives there? So click on "Review + create." And then click on "Create." Okay, that is done. Now I can simply search my resource group by simply saying "RG" and then I can simply say "Databricks." Oh, not "Master Class." Where's my resource group? "Databricks." Yeah, here it is. Simply click on this. And this is my resource group. Obviously, it is empty because we have not created anything.

So now, now, first of all, we'll be creating our Data Lake. Because Data Lake is obviously important. And without Data Lake, we cannot even do anything. Let's say if we just want to store our data. Makes sense. Because let's say you are processing your data, but at the end of the day, you need to store your data somewhere. Where will you store that data? Data Lake. Data Lake. Simple. Simple. Simple. Sorted.

So in Azure, we do not have Data Lakes. Anlama, are you kidding me, bro? Seriously? We have Data Lakes, but we need to create Data Lakes indirectly. Indirectly. Yeah. So by default, we create Blob Storage in Azure. But we create Data Lakes by selecting or let's say by configuring Blob Storage in such a way that it will be converted into Data Lake. Oamba, you just mentioned that life is already complicated. Why you're making it complicated? So, so sorry about that. So it is very simple. Okay? Simply click on "plus create." And then I will simply go to "Marketplace." Basically, this is the place from where we create our, uh, resources. All the resources will be created from Marketplace. Okay? Here you will find all the first-party Azure services or resources, second-party, third-party, fourth-party, all the party. Okay? I will simply search "Storage account." Because we do not have something called as Data Lake. We have Storage Account. Click on this. And then you will see so many storage solutions available within the Marketplace. But we need to pick Microsoft one. Click on this. And click on "Create." And this time, I will simply name it. See, this time, this resource group folder is already populated for you. Why? Because we created the resource by entering into the resource group first. And then we clicked on "Create." Common sense. "Storage account name." Let's give a good storage account name. I will simply say "Databricks ET." Makes sense. Um, "Databricks ET." Yeah, yeah, makes sense to me. It makes sense. And another, uh, quick note, you cannot pick this name now because I, I have taken this name. And your storage account name should be unique throughout the Azure network. Uh, so you have to pick another name. Okay? So now, "Region" is this one. I'm fine with this. And then "Redundancy." Simply pick "LRS" (Locally Redundant Storage) where your replica of the data will be, uh, replicated within the same data center. And it is the cheapest option. And if you are using a free account, then obviously it doesn't matter. Click on "Next."

So here's that thing that I was talking about. Basically, when you just check this box, you are telling Azure that, "Hey, I do not want just blob storage. I want Data Lake." Basically, Data Lake is built on top of Blob Storage. And by default, Blob Storage cannot be hierarchically, um, they cannot hierarchically store the data. No. So that is why we cannot create hierarchical folders within the Blob Storage account. But in Data Lake, we get this option. So that is why if you're working with big data, if you're just working, performing big data analytics, you need to just store hierarchical folders so that you can analyze the data. So that is why we call it as a Data Lake. And yes, Data Lake is the hottest topic. Yes, because everything is built on Data Lake nowadays. Okay? Click on this box. And click on "Review + create." That's it. And your Data Lake is ready. Click on "Create." And it will just deploy your Data Lake. And I think it should not take much time. I would say just a few seconds. Let me just refresh it.

So what's the second resource that we are going to create? What is the second resource? The second resource is Databricks. And before creating Databricks, we will simply set up our Data Lake. Why? Because bro, we going to follow Medallion architecture. So we need to just create three layers: bronze, silver, and gold. And obviously, we're going to create a container in which we will be just storing our data as a source. Makes sense. Good.

So now this is done. So now in order to go to your resource, you can simply click on "Go to resource." And it will directly take you to the resource. Okay? So this is your Data Lake. And if you are just seeing it for the first time, do not feel overwhelmed. Because we get basically four storage solutions within the storage account. The first one is "Containers," which is our Data Lake. "Fileshares," "Queues," and "Tables" are the other three services that we get along with the Data Lake that we do not use much. But yeah, you should know about these as well. Like "Fileshare" is just like the SharePoint solution in which you can just share the files with your entire team. And they can just use it. "Queues" is basically the solution similar to Kafka where you can just store the messages in a queue. And "Tables" is a solution that is similar for, similar to Cosmos DB, in which you can just store your data in semi-structured format in basically the JSON form. Okay? In the key-value pairs. That's it. But 99% of the time, whenever you're creating a storage account, you will simply use "Containers," which is the Data Lake. So let me just click on it. So this is the area. And within this, we create something called as "Containers." So simply click on "plus container." And I will simply create a container called "source." What is this source? So in this source container, we will put all our, um, you can say, source data Parquet files. Makes sense. Very good.

So I'll simply open this. And I will simply, uh, I will simply create directories. Yes. Click on "plus directory." And simply create three directories. "Orders." Basically, four. Click on "Orders." Very good. And then click on "Orders" first. Because obviously, we need to upload some data. So now I will simply click on "Upload." And I will simply upload the data from my system. Uh, uh, uh. So "Orders" first. That's it. Do not upload both the files, like "Orders first" and "Orders second." No, because we will be incrementally loading the data. So we will just upload it when we have our notebook set up. Just upload "Orders first." That's it. Okay, very good.

Now, just click on this "Containers." And this is the "source." Click on this. And let's create another directory. It's called "Customers." Very good. And do the same thing. Upload only one file, which is called "Customers first." Okay? Perfect. Now, create another directory called "Products." And then click on "Upload." And just upload "Products first." Perfect. Because you basically have four files. Okay? Three are the major ones. One is just like a small mapping file. That's it. And click on "source." And create your fourth directory, which is called as "region" or "regions." Click on "Upload." And then "regions." Perfect. Click on "Upload." That's it. Oops, I uploaded it here. So sorry. Let me delete it. Let me just go inside this. And then click on "Upload regions." Perfect. So this is uploaded. Okay, very good. Very good. Very good.

So now our source is set up. Source is set up. Now we do not need to do anything else. We will simply create empty containers. And those empty containers will be called as "bronze," "silver," and "gold." Okay? Simply click on "plus container." Because we are following Medallion architecture, so each container should be isolated from the other. So that's why different container for different layer. Okay? So perfect. Everything is set up. Very, very, very, very, very, very good.

Now, simply go to "Home." And simply search "Resource group." And simply search your resource group. Click on this. And you will see your storage account is here. Very good. Now let's create our Databricks. Because that is the thing that we're going to use in this entire project. Because I know we going to learn so much in this Databricks course, Databricks project, basically it is designed in such a way that you will master each and everything. Orchestration, pipelines, PySpark, silver layer, gold layer, slowly changing dimension, and blah, blah, so many things. You already know. I have just mentioned all these things. And trust me, you want to master these things, you're going to master these things. Let's create for "Databricks workspace." And let me just search, bro, what, what do you want, man? "Private offer management." Simply type "Azure Databricks." And just pick this one. Don't worry, we will just create this one as well. Why? I will just let you know. First, create a Databricks. Okay? Click on "Create." And you will feel the difference between hobby project and real-time or real-world project. You will feel the difference.

Then "Workspace name." Workspace name, let's say, let me pick, um, "Databricks." Makes sense. Okay? Yeah, makes sense. "Region" is "UK South." And "Pricing tier." This is important. So basically, I know you are using free Azure account. If you are not using free Azure account, obviously, why don't, why you want to just spend some money if you want to just create this Databricks workspace for just for the learning purpose, right? So instead of choosing "Premium," I will simply pick "Trial Premium," which is exactly same as premium, but it is just available for 14 days. That's all. And I think that's enough, bro. That's enough. Okay? Now it is saying "Enter managed resource group." Basically, this is an optional thing. And if you're just using Unity Catalog, this is not even required. This is just required to handle or to manage your clusters, to manage your virtual machines. That's it. And in the world of Unity Catalog, rest of the things will be managed by our own location. Okay? So it is optional. So if you do not provide any managed resource group name, it will automatically create any group. And we are fine with that because we are not going to monitor that group for this project. So it is fine. Okay? It is fine. Then click on "Next." And then click on "Review + create." And then simply click on "Create." That's it.

So it will take some time because it will just deploy all the things and create your Databricks workspace. It will take some time. But we have some work to do meanwhile. And let me just tell you why I was saying that we're going to create the other resource as well. Let me just show you. Let me just go to the resource group. This one. Yep. So basically, this is our storage account. And just imagine that we have Databricks as well. Okay? Makes sense. Makes sense. Very good.

So basically, let's say this is your Databricks. This is your storage account, Data Lake. Now, this Databricks wants to read some data from your Data Lake. Makes sense? Yes. How this will read the data from this Data Lake? How, how both are different resources? Both are different resources. How? So the answer is, it cannot read the data from Data Lake. So, Anlama, why did we create this? So basically, we know that it cannot read data, but we will allow this to read the data. Okay? So we need to provide the permission. How? Now, the thing is, if we would be using Azure's own resource, let's say Azure Data Factory, let's say Azure Synapse Analytics, then we can directly provide the access to the resource to use the Data Lake. But Databricks, I know, is a first-party service provider in Azure, but it is not owned by Azure. Databricks is a separate company. So now we have something called as "connector." So this connector is specifically made to to allow the access to this Data Lake. So that this connector can be installed, or let's say embedded, inside this Databricks. And once it is embedded, that means we can easily access the Data Lake with the help of this connector. That's it. So let's create that connector. And let's allow that connector to use this data. So simply click on "Create." And simply search "Databricks." And then you will see "Access connector for Azure Databricks." Simply click on this. And click on "Create." And then we need to simply provide the name. And I can simply call it as "Databricks ET connector." And I have so many connectors. I didn't delete those connectors. I will simply delete those connectors. Don't worry. But yeah, for now, it is fine. Click on "Review + create." Create. And then click on "Create." That's it.

Okay, so it is done. As you can see, like deployment is in progress. But yeah, Databricks is done. Oh, this is also done. It was quick because it is just a small resource, not a big deal. That's it. Go to "Home" from here. Or you can simply click on this "Resource group" as well. So you have all the options. So now let me just refresh it. And I should see that connector as well. Yeah, perfect. I have three resources. And it looks good. One is Data Lake, second is Databricks, and third is the connector between these two. Simple logic. Simple, simple, simple, simple logic. Very good.

So now, without wasting any time, let's actually go inside our Databricks workspace. Let's go inside this. So just click on "Launch workspace." And it will simply ask you, "Hey, login." Simply pick your email account of your Azure. Okay? Okay, okay, okay. H.

So this is the Azure Databricks workspace. And I know it looks amazing. This new UI is amazing, amazing, amazing, amazing. Now, I know that in this project, we're going to master each and everything available for us. For us means for data engineers. So obviously, we're going to cover all the things in detail independently. But just to give you a quick overview, let me just give you that. So this is the main thing towards left. This is the pane. So "Workspace." Workspace is basically your repository for all the things. Your notebooks, your ETL pipelines, your workflows, everything. Just like a folder. Okay? That's it. "Recents" tab. I think this doesn't need any kind of explanation. Okay? Then "Catalog." This is really important. Catalog is the area where we register all the things. Our catalogs, schemas, tables, functions, functions, yes, functions, volumes, volumes, yeah, volumes. Don't worry, you will learn everything. So yeah, all these things are registered under catalog. And catalog is the area where we can actually see all those things. Okay? Done. "Workflows." Workflows is the ETL capabilities that we have within Databricks. Can we do that? Yes. That's why in this particular project, I didn't pick Azure Data Factory because I wanted you to learn ETL capabilities within Databricks as well. And it is really, really, really important. Okay? Then "Compute." Compute is the thing that you use to process anything, or, or I would say, everything. Compute is basically the cluster that you create. Your Spark cluster, your executor nodes, driver nodes, everything. It is here. And you can easily create compute by using, um, drag and drop feature, or by just using a simple UI. That's it. You do not need to use "spark-submit" again and again. No. Then "Marketplace." Yeah, Databricks also has a marketplace from where you can just use the resources. That's it.

Okay, then this area is newly added. And this is majorly focused on SQL Warehouses. Like once you create your warehouse or lakehouse, then you can actually connect that lakehouse through external endpoints. And you can simply share that link with other data analysts or data scientists. And whenever they will be querying the data, those queries will be optimized through this area, which is called "SQL Warehouse." And it is really, really, really amazing. And in the SQL Warehouse, they have launched their own on, own cluster, in which they have optimized those clusters to run SQL workloads only. So when you'll be just querying the data using that cluster, using that compute, you will be getting amazing results, or I would say, much faster results as compared to job cluster or I would say all-purpose cluster. So those clusters are specifically built for running your SQL queries. Makes sense. Very good.

And by default, we get something called as "Serverless SQL." Serverless SQL compute, which is the same compute that we're going to use in this particular project as well. Because it is already created for you. It is not much expensive. It is lightweight and enough for our transformations. Yes, we'll be creating some special computes as well in this video because we're going to create Delta Live Tables and so many things. So we know that. But just to get started, we can easily use some these kinds of clusters. Makes sense. Makes sense. Makes sense. Makes sense.

So this is "Data Engineering," where we have "Job Runs," "Data Injection Pipelines." And "Machine Learning" is totally, you know that, all your, um, machine learning models, machine learning building, everything is in this area. And obviously, we are not covering this machine learning thing. Okay? So simply click on the "SQL Warehouses." And within the "SQL Warehouse," you will see "Serverless Starter Warehouse." And this is the, you can say, compute that is already there for you. Okay? And if you just click on, and if, if it is not here in your case, you can simply click on "Create SQL Warehouse." This is something that it will create a SQL compute, or let's say, compute for SQL workloads. Okay? Then you have "All-Purpose Compute." This is a compute that we create traditionally. Okay? Then we have "Job Compute." Job compute is something that is used for production scenarios, or if you are just building Delta Live Tables. Okay? Then "Vector Search." So this is newly added. And this is a kind of vector DB. And this is used by data scientists to perform vector searches. And it is kind of a data in which, um, it gets translated into multiple dimensions. So you do not need to worry. It is not your area. So do not worry. Then "Pools." You can even create the cluster pools in which you can define, "Hey, this is my pool." Pool is basically the set of, uh, machines that are available for you. And you can simply set, "Hey, this is the number one machine, like minimum number of machines that I need all the time." So it will be always up and running for you. And these are the maximum number of machines. That's it. Okay? Then "Policies." These are the cluster policies. The very famous one is "Personal Compute." Okay? And these are the pre-built policies. So, um, let's say "Personal Compute" is the one in which you do not get much things to configure. And if you just go to "Job Compute," then you will be seeing some fields already filled for you. And you can even create your own policy as well. So do not need to worry. And as you can see that "Shared Compute" as well. So these are like, uh, these four are for like all-purpose. And this is just for the job cluster. So this is the most popular one in which you just need to click on a button. And all the things will be filled for you. And then just click on "Create." That's it. That's it. That's it. And this is the "Apps." Okay?

So now, enough, enough, enough overview is provided. Now let's get started with our workspace. Click on this. Okay? And then click on "Create." And then click on "Folder." And then just provide a good folder name. And I will simply say "Databricks ET Project." Okay? Click on "Create." Within this folder, we

The notebooks we create will all make sense. Very good. Now, let's start ingesting our data. Let's do that. We know that we are going to ingest just our data, and we'll be doing it incrementally. Yes, that's true. But I told you that we have one mapping file that is fixed. We are not going to pull it incrementally. So, how can we just ingest that data? And I'm going to show you an amazing feature introduced by Databricks, and I love that feature.

Basically, in this particular feature, you do not need to write any code to pull any data. What? Yes, you do not need to write any code. It is totally no-code. What? And let me just tell you one more thing, bro, hold on. With the help of this, you do not even need to create any table. The table will also be created for you. This is an amazing feature. I love this. Okay, I know you are really excited to know about this feature, and similarly, a lot of these kinds of features will be discussed today. Don't worry, just hold your emotions.

And before that, let me just tell you the most important thing, because even before getting started with anything, we need to set up something called as Unity Catalog. Unity Catalog, you need to set up. Okay, and how can you just do that? First thing, you need to simply go to Databricks Admin Console. And how can you just go there? Click on this dropdown, and you should see something called as "Manage Account." Okay, sometimes you will not be able to see this. Why? Because this particular "Manage Account" option will be visible only to the administrators. And you are not an admin if you are not seeing this. What? I am the only owner of this account. Why can't I see it?

So basically, the thing is, just click on this. This is your email ID that you are using, right? But do you know what is the actual email ID that Azure uses? It is like it ends with something called as #ext. Let me just show you. Click on your Azure, go to Home, then search "Microsoft Entra ID," Entra ID. Bro, click on this, go to Users, and you will see that this email ID has the principal name. This. So this is the original email ID that it uses. Okay, so when you will just click on this, you will not be able to see this "Manage Account" if you are not able to see this. Okay, so what's the solution for that?

The solution is very simple. Simply go to Google and simply search accounts.databricks.net. This one: accounts.databricks.net. So simply hit enter, and it will simply ask you – obviously, it will not ask me because I am already logged in – it will simply ask you, "Hey, simply put your email ID here." Listen to me carefully. Listen to me carefully. Okay, now when it will be asking you to provide your email ID, you do not need to provide this email ID: normal@gmail.com. No, simply provide this long email ID, which is your user principal name. Then you can simply put the password, and it will, I think, ask you to just, "Hey, provide reset password and all." Do that. Then you will be able to land here in this console. And let me show you something. Click on this. You will see my long email ID. See? So I'm not logged in with my normal Gmail account. No, I am logged in with my long email ID.

Okay, Anchal, hold on. If you are not logged in with your normal email ID, how can you see this "Manage Account"? I will tell you, baby, hold on. So first of all, just do this and just land on the console page first of all. Okay, simply land here. So once you land here, then what I did, I just went to User Management, because this is the top level of the hierarchy of our user management. I went there, and then I just added my email account, and I just added the user. Click on "Add User," "New User Email," just put your email ID. "New User Full Name," just give any name. Obviously, it is your name. Okay, click on "Add User." Then I assigned it as "Account Admin." Roles assigned as "Account Admin." I think there are only three roles. I can just check. Let's say hello.bro@gmail.com, and then click on "Add User." And then it will simply ask me roles. See, there are three roles: Account Admin, Marketplace Admin, Billing Admin. So I just created myself as Account Admin. You can obviously create Marketplace Admin as well. And I would say that's not mandatory. If you would say, "Hey, if I just want to add any user and if I do not want to make him the admin," obviously, it's up to you. It's not like you have to make the person as admin. No. So you can simply say "Last Name" and "Roles." Obviously, see, I have not given any role. I can simply say "Save." That's it. And this user is saved here. See? So now this is just a user but with no privileges. But obviously, if you just want to make yourself as an admin, simply pick "Account Admin." Let me just delete this user because otherwise, next week I will be thinking, "Hey, who is he or she?" Click on these three dots, "Delete User," "Confirm Delete." Who is she? Okay, sorted.

Now let's talk about the main thing. Go to Workspace or Catalog. Just go to Catalog. Okay, now we need to create something called as Unity Metastore. So whenever we just want to enable Unity Catalog for our Databricks workspace, we are like, we have to first enable the Unity Metastore. Unity Metastore comes at the top. Let me show you the hierarchy, bro. Let me show you the hierarchy. So Unity Catalog. Oops, let's type it. Unity Catalog. And just search this one: "What is Unity Catalog?" So this is the hierarchy. See, at the top, we have to create our Unity Metastore. Then we assign our Databricks workspace to that Unity Metastore. When we do that, that means we are enabling Unity Catalog. What is the advantage of it? Basically, when we allow Unity Catalog, that particular catalog will be shared with other Databricks workspaces as well, if we allow, if we allow. Okay, don't worry about this thing much for now. You will understand all the things. What is Unity Catalog? What I am talking about? Just be with me. Just be with me. Just be with me, like, just be with this video. Just be with. Okay.

So now, Anchal Lamba, who created these three metastores? Me. So who will delete these? Me. But I will not delete it now. Okay, so just forget about or just ignore these three. So Anchal Lamba, who created this one? Azure Databricks created this for you by default, because whenever you create your Databricks workspace, you will get one Unity Catalog for free, or basically, Unity Metastore for free. For free. How? How costly it is? Obviously, it is free, but yeah, you do not need to create this. And I got like free things. I will simply say, "Hey, click on this and click on these three dots, click on delete." That's it. Why? Because I want to create my own, bro. I will simply copy the name of this. I'll simply set delete. Perfect.

Let's create our metastore, because obviously, if you're sitting in the interviews, the person will say, "Hey, you are creating this. What will you see? And how can you just delete the metastore? How can you just create your own metastore?" You should be prepared for all those questions, right? And with confidence, with confidence. Confidence is very, very important. And just be overconfident in your interviews, and just mark my words, bro. Whoever is taking your interview, I know the person is very knowledgeable, but that person also doesn't know everything in this world, right? It's fine then if you do not know that thing, but just be confident. Just be confident. Just be confident with your knowledge that you know, and just learn all the ins and outs of the topics, and just be confident. Just say like, "I'm the master of this skill, bro. Just ask me anything." And obviously, I can be answering 99% of the time, but obviously, in 1% of the cases, I would think, and I would simply say, "I cannot remember this thing, but I have just worked with this thing or similar to this thing." That's it, bro. Just be overconfident, and do not hear to anyone. Just be overconfident. That's it.

Just click on "Create Metastore" and click on "Name." So now we need to create our metastore, and obviously, we need to provide the name to this metastore. So I'll simply say, um, let's say "Databricks ET Metastore." Okay, very good. Region, I will simply pick "UK South." So another interview question: "Person will say, 'Hey, there is a metastore in UK South, okay, and you need to create a metastore. How can you create it?'" You will say, "Um, we can simply click on 'Create Metastore' and we can simply do this." He will say, "Hey, bro, just cut the call, and you are rejected." Why? Because the thing is, you can only create one metastore in one region. So if a person is saying, "Hey, there's a metastore already in that location," simply say, "Either delete that or choose another location," because you cannot create two metastores in one location. So this can be a tricky question. You should be prepared, and I think now you are prepared.

Okay, and by the way, just hit the subscribe button right now, right now. I was just looking at the analytics of my channel. I was like, "Why people are not subscribing to my channel? Why, why, why? Is there any reason?" I know there are some haters, but I don't want to give a reply to them. But I know you love me a lot, so simply click on "Subscribe" button. I know you would have forgotten to hit that at that time, so simply hit it right now and just share this channel with others as well so that they can be learning a lot of things. Okay, so simply now, I know that you have done that.

So now let's talk about this other option, which is called ADLS Gen2. What is that? And if it is optional, Anchal Lamba, should we do that? Yes. What is this? Basically, Unity Catalog needs some location in which it can store your managed data. Just tell you one thing: whenever you create a managed table, you know that data will be managed by the Databricks. Okay, where will it store the data? Databricks is not a storage solution, so where will it store the data? It will store that data in this location which you'll be providing here right now. Okay, so simply provide that particular location, and then you can easily work with that particular location.

So now let's provide the location. So container name is, we need to create, because we do not have any container. So simply go to your Azure, go to Home, and then just go to your Databricks workspace, go to your Data Lake, and then click on "Containers," and then simply create a container. And I'll simply say "metastore," because this is the container that is only and only available for metastore. And I will simply add a tag: DNT. Do Not Touch. DNT, not D&D. So this is done. Metastore. Okay.

Basically, these kinds of errors come, um, when we like did not define the external location, um, precisely. But that's not the case because we can obviously read this data. We have just read this data, and I can just rerun this. Uh, let's see if it is running.

[Music]

Let's see if it is working fine. Yeah, it is working fine. So, data is there, and location is there. The error is something else. Let me just read the error for one more time. Cloud files, cloud files, do format, schema, location. Okay, first of all, we should not create schema location in the silver. We should create the schema location in bronze. So, it is not aligned with the error. Yeah, I still, oh, I see the error. It is not load, bro. It is not load. It is, I think, yeah, it is load. It is load. It is load. I thought we are just writing that. Let me just see. I got the error. I got the error. I got the error. Just a typo. I have written windows instead of Windows. Just a tip for you. If you see this, like because I know these kinds of errors only come when you have something wrong either with your, um, external locations. Maybe you have just provided the wrong location in that, or there is some typo in your URL. Like, both things are not matching. That is the only issue that arises. If you have like, like that is the only resolution, uh, if these kinds of issues ar, uh, these kinds of issues arise. So, I can simply correct this, and I can simply run this. And it should work because I know my external locations are fine. Yeah, perfect. See, perfect, bro. Perfect. Yeah, perfect.

So, these are basically, you should be able to debug your errors, okay? And obviously, if you see errors, feel happy. Do not feel sad. Feel happy because there's something new to learn. Maybe you have made some mistakes. Okay, let's correct those mistakes. And let's not, obviously, feel sad if we see the errors in the future because I have already seen like these kinds of. So, I was happy. I was like, okay, I know the resolution. I just need to investigate. That's it. So, just build this habit. Do not be like a crybaby. What is the solution of this error, bro? You are a data engineer. If you are, obviously, you will be landed as a data engineer in the company. And if you will see some errors there, what you will do? Will you go just go somewhere, somewhere, and will just type, "Hey, what is the solution?" You will find it. Spend two days, three days, four days. Who is judging you right now? You're learning. What's your issue, bro? You have all the AI applications, ChatGPT, and everything. What are you doing, man? Just try to debug on your own.

Okay, good. So, now we have just initiated our Spark query. Okay. So, see, I have also went to like ChatGPT and I also just provided my code. I, I, I just asked it, "Hey, what is the wrong? What, what is wrong with this code?" Because everything is fine. Then it just pointed out this thing like "Windows" because obviously, when you're just building and writing so much of code, you cannot look at each and each and every alphabet. But you can just leverage these tools like this. Okay? Just, just say that, "Hey, is, is there anything wrong with my code?" If it says no, then it's your duty, being a data engineer, to troubleshoot the issue. If there was nothing wrong in my this code, then I knew I just need to go to my external location and just need to double-check my URLs. That's it. That is the, you can say, steps that you need to follow whenever you see the errors. Solve it step by step. Because one, the first step is like, "My code, is it fine? Is it syntactically fine? Is there any typo?" I don't want to spend my time into just checking each and each, like each alphabet. I will simply submit my code to AI. "Hey, simply just give me any typo." If there it is, it said, "Hey, you have just written 'windows' instead of 'Windows'." I was like, "Okay, my problem is solved." I do not need to go to, um, external location and check the URLs. Solve all the errors like this, bro. Okay? Simple.

Now, data reading is done. Now we need to write this data. Okay? And how we can just write the data? I can simply say, uh, `df.writeStream` because obviously, this is a streaming. Then `.format`. I will create Delta format. Why? Because I love Delta format. Just kidding. We will just create Parquet file format because in the bronze zone, there's no need to create Delta formats. I will simply say `parquet`. Okay? Then I will simply say, `.`, um, `outputMode`. Okay? And `outputMode`, I will simply say `append` because obviously, we should just pick `append`. And it has automatically taken `checkpointLocation`. And `checkpointLocation` is this one. Yes, I want to create this `checkpointLocation`. Perfect. Yes. And as I just mentioned in the explanation as well, this location should be matching with the, uh, schema location. This one. So that you just need to manage only and only one folder. That's it. Okay? Very good.

Now, we need to simply say `.option("path", ...)` what is the path? Let me just copy it. And I will simply create `orders` folder and in the bronze container. Makes sense. Makes sense. Now, let's talk about `trigger`. We need to trigger this, right? Because this is a streaming. I will simply say `once=True`. Why? Because there are so many options. I can even write, let's say, uh, `processingTime="10 seconds"`. Let's say. So, what it will do? It will keep on running after 10 seconds automatically. In, like, in every 10 seconds, it will be running. So, if it is the requirement, do it. But in my particular case, I don't want to run my cluster for every 10 seconds because obviously, it is costly. What I will do? I will simply run `once=True`. What it will do? If I say `once=True`, if I say `once=True`, it will simply read all the files which were not processed before, and it will simply perform that particular, uh, you can say, data writing part, and it will simply stop the streaming query. It will not keep on running. It will not keep on running. It will simply load the data. It will simply write the data. And that's it. This is the best thing.

Let me now just start this query. Let me just do this. And I know in my, see, streaming initialization. So, now streaming is being initialized. You can simply click on this drop-down and this one as well to see the graph. So, this is basically the streaming graph. And obviously, you will not see the continuous graph because this is `once=True`. So, it will first go up, and that's it. Footage of now loading the data, and it has done the job. Why? Because stream is inactive. That means it returned the data. It was so quick, and it stopped. Now I will just take you to the data lake containers, bronze. This is my checkpoint. Orders. Told you. Very good. Go to orders. This is my data. This is my data. Very good.

Now, if you just go to bronze, go to checkpoint. What is this checkpoint location? Open this. You will see all the things. This is schemas. This is your schema location. Okay? Go to commits, and then just go to offsets, and then just go to sources. Perfect. In zero, you will see RocksDB. This is the folder which maintains your current state and future state and responsible for exactly once processing. Okay? Now, your data is successfully loaded to bronze. Only one data. I know, only one data. Okay? Now, it is stopped. So, you do not need to worry at all. I can simply click on this close button.

Now, in order to confirm how many records do we have, let me just apply a quick data reading here. And I can simply say `bronze`. Okay? And I will simply say `orders`. Obviously, uh, `orders`. Yeah, `orders`. Okay? Let's run this and let's see what do we get. Because we know that we have 9990 rows. Makes sense.

Now, what I will do? I will simply go here in my bronze container. Okay? And in the bronze container, or sorry, in the, in the, in the, in the source container. The source container. Now, let's say next day, you are receiving `orders_second.parquet`. Which means, like, the number of orders for the next day. Now, I'll simply say upload. Upload `orders_second.parquet`. Make sense. Perfect. Now, this data is uploaded. `orders_second.parquet`. Makes sense. Very good.

Now, your notebook is scheduled to run this, or let's say, process this data. Okay? Now, let's go to our Databricks. Now, let's rerun this query and write the data for one more time. And let's see what happens. Obviously, stream is initialized. Makes sense. And it is done because it was really, really quick. And you cannot even see. And once it is done, it is stopped. Why? Because of this reason, `once=True`. Okay? Makes sense. Because obviously, if this is not required to run your cluster 24/7, why would you be running it in just every 10 seconds?

Okay, now let me show you something else. Just run this command, and you will see the, the, the magic. Let's scroll down. And how many rows that you can see? 10,000. How many? 10,000. Why? Because earlier we had 9990 rows, and the second day we got, got, we got only 10 rows. So, how many rows, or let's say, how much of data it has read? Only 10. Only 10. Not both the files. Not both the files. And if you just want to confirm it, I can just show you. There's a better way of it. You can simply say `df.count()`, and you will see the actual number. Okay? How much, how much it is? 10,000. Because sometime it by default shows 10,000. You'll say, "Hey, Lamba playing smart, huh?" No, bro. 10,000. That's it. Instead of 9990 + 10, again, and it will be equals to 19,990. No, only 10,000. Even, even this query is stopped. Till it will only process the new files. Why? Because RocksDB. RocksDB folder is actually capturing your all the information. And even if this query is stopped, that doesn't mean that this query is like reprocessing all the data. No. So, this is your interview question, bro. Even if the query is stopped, new data will be processed. Only the new data. Reason: RocksDB. Makes sense. Makes sense. Makes sense. Makes sense. Very good.

Now, you understood the concept. You have seen the picture. All the things. Now, let's make this notebook dynamic. What do you mean by dynamic? Dynamic, dynamic. Let me just tell you what I mean. I know that I have total three particular folders. Yes, as source, orders. Yes. And then, uh, your customers. And then products. Makes sense. Yes. Just imagine in the real world, you have hundreds of folders. Will you be creating a new notebook for all the 100 tables? No. Will you be writing the code for all the 100 tables? No. What you will do? You will create a dynamic notebook. Okay? And you will be parameterizing the notebook. Okay? And you will be using loops for that. And using workflows. Makes sense. Makes sense. If not, don't worry. It will make sense now.

So, now let's do one thing. First of all, let me just make this beautiful notebook. What? Make this beautiful notebook. Make this notebook beautiful. What? Oh, man. Okay, let me just write, um, data writing. Because obviously, I'll be uploading this notebook. You should feel like, "Oh, this is provided by Anish Lamba." Okay. Data writing. Perfect.

Now, what I will do? I will simply create some variables. Makes sense. Variables, or you can say parameters, basically. Because I will make, see, there's a trend. If you closely observe, if I just want to read data, I have only this thing, `orders`, which will be changing for products, it will be a `products` folder. For customers, it will be a `customers` folder. That's it. And `checkpointLocation` will be the same for all the, like, different. But yeah, it will be exactly same as your name of the file, right? Perfect. Same thing with data writing as well. I want to write my data for the orders. So, what I will do? I will simply create a variable called, let's say, `data` or you can, you can say `file_name`. Simple. `file_name` or `folder_name`. Simple. Or `source_name`. Anything. Okay? So, let's do that. And how we can just do that? Basically, you need to first of all remove this cell. And let's create a text. Beautiful text. And let's say, "Parameterization" or "Dynamic". "Dynamic Notebook" or "Dynamic Capabilities". Makes sense. Perfect. Dynamic capabilities.

Now, we will simply say `dbutils.widgets.text()`. And for that, I think we need to import `dbutils`. I will simply say `import dbutils`. No, it is already there. Okay? Because it is something similar to `shutil` in Python. So, in Python, we just import `shutil`. `shutil`. Simply say `dbutils.widgets.ext()`. It will simply create the widgets. Okay? Hm. Now, I will simply say `file_name`. And by default, I will simply say `orders`. By default. Let's run this. You will see one box here. Wow. Similarly, I will create two more, or basically one is enough. Yeah, one is enough. Because let's say I want to print the value of this. I will simply say `dbutils.widgets.ext()`. Oh, sorry, not `text`, `get()`. I simply say `get()` and `file_name`. So, what it will return? Very simple question. `orders`. Now, let's change the value of this. Let's say `customers`. So, what it will return? `customers`. Wow. Then let's say `products`. `products`. So, it will be changing in the real time. That's what I want.

So, what I will do? I will simply first of all create this. Okay? And there's another way of creating this as well. You can simply, like, click on plus and you can create more if you want. And you can even delete it if you do not want. Okay? And I think there is another button here from where we can just create. See, "Add parameter". And you can do the same thing here as well. But I like to just create it using widgets. And it is a better way to do that. It's up to you. Like, bro, it's up to you, bro. Come on, man. Come on.

So, what I will do? I will simply remove this value because I do not want to provide any default value. Let's run this. And you will see that whatever value I will be just providing here, it will just take that value. Perfect. And we will just dynamically pass this value. Okay? So, now what we need to do? I will simply store this value. And I will simply say `file_name`. Makes sense. Or let's say `p_file_name`. Just for the readability. And now I can store this value in `p_file_name`. And I can actually return this using this. See, now what I will do? I will make my code dynamic. Because I'll be using this value in the runtime. I will simply go here. First of all, here. I will simply use f-string. So, this is a common string that we use in Python. If you just want to combine your string with variables. Some people do like use format function. I hate it. I personally use f-strings. I love it. I will simply say `file_name` or sorry, `p_file_name`. So, this way we can actually combine text and variable. Okay? Same thing with here as well. Because we want to just read `orders` or whatever the file name will be. Okay? Perfect. Makes sense. Very good.

Now, data writing. Everything is same. Let's change it here. So, this is just a design question. A kind of, like, this is just a kind of design that you should be aware of. If anyone is asking you, "Hey, why did you pick this?" Because obviously, when you are just explaining the projects in the interviews, you should be very well-versed like, "Why did you pick this? And why didn't you pick the other thing? What is the advantage of this?" Okay? You are just showcasing your project. Hello, sir. Hello, ma'am. This is my project. I did this, bro. Why did you pick this? Why just say that, "I do not want to write my code for, let's say, thousands of tables." So, that's why I have created a parameter that I can change in the runtime. And I don't want to write my code again and again. Just give a reply like this. "Hey, sir, this, what is this, bro? What is this? What is this? I told you, be overconfident, bro." Perfect. Perfect. Perfect. Now, it is fine.

Now, if I want to run this, what I will do? I will simply run this using `run all` command. Let's do this. And you should know what it will do. It will run all the cells. What it will do here? The answer is nothing. Because we do not have any new file. Makes sense. We do not have any new file. We can just confirm it. Go here. Go to containers, bronze. And there should be only two files. Oh, wait, wait, wait, wait, wait, wait. What is this? Oh, man. Wait. Do you know what it has done? Do you know we didn't add f here? That's why it took that as string. Okay? So, just a small typo. Let's remove these folders again. These do not, do not feel sad if you just see these things. Feel happy. Feel happy. These are directly aligned to your learnings. Okay? Now, let's do `run all` for one more time. Okay? Tick, tick, tick, tick, tick. And what it will do? I have already told you, nothing. Because we just have zero new files. So, we should just see two, two, two files. That's it. Orders and just two files. Refresh it. Still two files. That's it. Still two files. So, this is the thing. And what is a Spark metadata? Basically, whenever you just perform `autoloader`, it just holds a metadata of the individual file as well. So, do not need to worry. The main thing is your, um, RocksDB and schema location. Okay? Okay. Okay. Okay. Okay. Okay. Sir. Okay.

Now, it's time to actually create the pipeline. So, that we can run all the things. Because you, you will say, "Anish Lamba, come here, come here, come here, come here, come here." So, oh, just call me like this. I will give you a reply. Okay? Yeah, there are a few people who can just call me like this, but not everyone. Like, obviously, my mom can call me. I would love to hear that. Okay? Wait. Now, you will say, "Anish Lamba." Now, if I just want to process hundreds of tables, will I be writing this one by one and clicking on `run all` for one by one? No. You are my data fam. Okay? Let me tell you the best way to do it. What you will do? Simply go to workspace. Create a new notebook. Okay? And call it as parameters. Okay? Just define an array. H. Okay? I will simply say `datasets`. Okay? Array. And then define dictionaries. I will simply say, uh, `file_name` is `orders`. Perfect. Then `file_name` `customers`. Okay? Then `file_name` `products`. Very good. So, this is my array. If I just run this, it is attaching it to the cluster. Okay, baby? Attach it. So, this is my array. I want to use this array as the input for. Don't worry. Don't worry. Don't worry. You will learn everything. I'm just giving you a higher-level overview. So, we will use this array because I just mentioned this so many times, so many times, that we're going to use for loop. If you are coming from Azure Data Factory background or Azure background, you would know "For Each" activity, right? This is exactly same. And if you're not coming from Azure background or Azure Data Factory, just ignore what I said. Okay? Do not worry.

So, basically, I want to use this as my input. As my input. How I can just do that? We have something called as `dbutils.jobs.taskValues.set()`. Yep. And then whatever, uh, name I'll be providing, I'll simply say, um, `datasets`. Makes sense. Yeah. `datasets` or `dataset` or let's say `output_datasets`. It will make more sense. `output_datasets`. And then I will simply provide the `datasets` variable. And obviously, encode it. So, what we are trying to do? This particular utility, this one. Whenever you'll be running this particular notebook, it will be running this notebook in the workflow that I'm going to tell you right now. Whenever we'll be running this notebook in the workflow, with the help of this utility, which is `dbutils.jobs.taskValues.set()`, we can return this `datasets` with this particular name. So that we can use this as the input. Input. Just try to imagine, bro. I know you're really good at imagination. Just try to imagine. Just try to use your superpower. Simply run this. By the way, there's no need to run this. But you can run this in order to confirm everything is fine syntactically. Now, let's go to workflows. Now, let's create our first job. Create job. And then I will simply say, so this is our job. Okay? Now, I will simply name it as `bronze_incremental`. Makes sense. Very good.

Now, this is the first activity that we need to perform. And this activity is our parameters notebook. Simply click on this one. Type is notebook. Okay? And click on this. And wait. It is hanged. Or, or, or what? What tasks? H. Okay. Task name. Workspace. Oh, man. It is hang. Robot. Oh, man. I think I need to just refresh. Let me just do a refresh. Quick, quick refresh. Okay. Perfect. Yeah, it was hanged. So, task name is, let's say, `parameters`. Because this is just a parameters file. Notebook, obviously. So, in this, we can just perform any activity, such as Python script, Python wheel, SQL, pipeline. What is pipeline? You can actually run the whole pipeline as an activity as well. Don't worry. This we will discuss at the end of this project, where we'll be just creating end-to-end flow at the end. Okay? For now, we are good with the notebook. So, where is the notebook? M. Perfect. You can simply search notebook. And done.

Now, you need to pick the path. And path is your Databricks ETL project folder. And parameters file. Confirm. Now, compute is, um, cluster. Like, all, all job cluster or all-purpose cluster? Because I think it is better to run it. Then, do we have any kind of parameters here? No. In this, we do not have. But in the other notebook, we do have. That we need to feed it through this notebook. Makes sense. And just remember, we need to keep the exact parameter name. Exact, exact. And I, I know that the parameter name is `file_name` in our Databricks notebook. For now, it is fine. Simply click on `Create Task`. And perfect. This is the task created for you. This is a task created for you. Now, I'll simply say `Add Task`. Okay? Another notebook. So, this is already linked to it. And I'll simply say `bronze_incremental`. Or let's say `autoloader`. Makes sense. Very good. Then I will simply say path is this one. Confirm. And compute is obviously this one. And now parameters. Yes, we have parameters. Simply say parameter name `file_name`. What is the value? Now, is the thing. Because we cannot provide. You will say, "Hey, it is so simple. We need to just pick the output of this and we need to provide it here." No. Because that is an array. We cannot provide an array to a parameter. No. We can just provide only one single value. So, what we need to do here? We will simply say, "Hold on." We will provide this value. Hold on. How? Simply click on `Create Task`. And then click on this. Fit to viewport. Viewport. So, that you can see everything. Now, click on this. And simply click on this loop over this task. This is "For Each" activity. Click on this. Now, this is a loop. Now, it is saying, "Hey, what is the value of the loop?" Now, you will pick that array. And you will feed that loop. Click on this. Okay? And then input is dynamic. So, this is the dynamic content. Click on this. And you can simply write your code. And it is already giving you suggestion. `task.parameters.value`. And my value is what name did I give? I think I gave `output_datasets`. `datasets` or `dataset`. Let me just check. Duplicate. And go to recents. Because you need to pick the exact name. Okay? And it is `datasets`. Yeah, perfect. So, now this is the value. So, every time it will first go to this value. Okay? Then it will go to this, this value. Okay? Then it will go to this value. Perfect. Perfect.

Now, let's say it's done. Now, just click on this inner activity. Okay? Expand it. And now you will pick this value. How? Dynamically. Click on this. And then you will simply say `input`. What is this `input`? `input` is this dictionary. `input` is the item within the array. And what is the item? Item is this whole dictionary. What is the second item? This dictionary. What is the third item? This dictionary. So, every time it will just pass the whole dictionary. And we know how to fetch the value of the dictionary using dot key. That's it. Dot key. So, what is the key? Obviously, what is the key name? `file_name`, bro. Okay? So, this way it will work. And you can simply say `Insert Dynamic Reference`. And you can see a lot of things here. `input.input.blah blah blah`. Simply click on this. And then `file_name`. Same thing. Perfect. This was all about your task. So, this is the pipeline that is built for `bronze_incremental`. `bronze_incremental`. Only on, only on. Makes sense. Or we will just rename it. Don't worry. We have the opportunity to rename it as well. So, this is the pipeline. And now let's click on `Save Task`. Okay? And now just click on `Run Now`. Let's see if we do have any errors. Because we didn't test this. Click on `View Run`. And let's see if it runs fine. And if there are any errors, it is fine. Because we didn't test this. We simply saved it. So, this way it will first run this parameters file. And then that parameter, that array will go to this particular loop. And it will perform that `autoloader_orders` one by one, one by one. And don't worry, we can just actually validate this as well. Because we have just copied only `orders` data, not the other data, right? Makes sense. Very good. Very good. Very good. I hope you are understanding this. Come here. Done. Lamba. What is this? Can you see this? Three scheduled. How many values did we have? Three. Very good.

Now, let me just take you to the data lake. Okay? Go to bronze. Wow. All the three folders along with the checkpoint location. Wow. Wow. Looking like wow. Wow, man. Wow. See, see. So, now, just a quick test. Anish Lamba, we trust you. We love you. And just for the sake of testing, let's test this as well. How? So, we know that in the products and in the customers, we just have one, one file. Let's upload one more file. And just to give you a quick overview, in the products, we have 490, uh, records in this one. And in the second file, we have 10 more records. So, in total, we should have 500. In the, uh, customers, we have, I think, uh, 1990 records in the first file. And 10 records in the second file. So, in total, 2,000 customers. Okay? Let's test it. Let's upload the products. First product. Second. Okay. Okay. Okay. Okay. Upload. And now refresh. Refresh, bro. We just uploaded. Where? Where's the file? Let's upload for one more time. Product second. Yeah, it is uploaded now. Okay? So, now, just do the same stuff with customers as well. Mhm. Open. Upload. Perfect.

Now, let's rerun this for one more time. And how we can just rerun it? Simply click on the workflows. This is the `bronze_incremental`. Click on this. And simply say `Run Now` for one more time. And it should only run the new files. And for the other table, it should not do anything. Because there's no new file for orders. So, it is double testing. Let's see. Let's see. Let's see. Let's see. As you can see, one scheduled. Okay? And in total, we have three. So, second is also scheduled now. And then third will be scheduled very soon. Third is also scheduled. So, see, this way you build pipelines dynamically. Perfect. Done.

Let's validate the results. Go to recents. Go to bronze layer. And let's read the data one by one. Okay? Uh, I will simply say `df = spark.read.format("parquet").load()`. And then, uh, perfect, perfect. Yeah, let's do this. First of all, let's read the orders data. And we should see 10,000 rows. What's the error? `p_file_name`. Oh, f-string. So, there should be 10,000 rows. Perfect. 10,000 rows. Let's see. Customers. Customers. I think in total, we should have 2,000. Not more than 2,000. Okay? Let's see. 2,000 total. Yeah. Simply scroll it down. You can say refresh. Now, see. Simply scroll down. And it will be refreshed. 2,000. Very good. Now, let's see the products. Products should be 500. I guess. Because there were like, total 500 products. Simply run this. And perfect. 500 rows. Perfect, man. Perfect. Very good. Very good. Very good. Our data injection bronze layer is done.

Now, it's time to actually create our silver layer with all the transformations and with all those crazy things that we're going to learn in this particular section. And let's see what do we have for our silver layer. Let's talk about our silver layer. Okay? Let's create our silver layer. So, first of all, I will simply go to. Let me just check if my mic is working. Yes, it is working. Because sometimes it doesn't work. And then I have to re-record everything. So, first of all, um, let's go to workspace. Okay? Okay. Okay. Okay. And then I will simply go to this workspace. And okay. And go to this folder. And then we can simply create our new notebook, which is called `silver_layer`. So, basically, there will be three notebooks. Because not three, actually four. Yeah. Because we have like four files. So, we'll be creating different, um, notebook for different, you can say, table. Because every table has its own transformations. So, there will be like four notebooks. Okay? Makes sense. I'll simply say `silver_orders`. Let's do this first. And let's attach this to our cluster. Makes sense. Okay. Perfect.

So, now, first of all, first of all, first of all, let's put the heading. Let's say, "Data Reading". Very good. Now, I will simply read the data. I'll simply say `df = spark.read.format()`. And format is `parquet`. Makes sense. Then I will simply say `.load()`. Simply because `parquet` doesn't need to define any schema. Because schema is already there at the footer of the file. I will simply say `bfs.bronze@databricks.tfs.core.windows.net`. Perfect. Let's see what do we have to transform in this. Obviously, you'll be using like PySpark functions and amazing functions. Unable to input schema for parquet. What? Oh, obviously, bro. What is the folder? So, sorry. Folder is `orders`. Perfect. Now, let's display the data. Perfect. Display. Okay. Perfect.

So, this is our data. And obviously, usually we do not have much stuff to do, um, whenever we just perform, you can say, transformation on top of fact table. Because fact table is full of IDs which we cannot transform and some, you can say, numerical columns. But when we use autoloader, you should know that we get an extra column. And this extra column is called `rescueData`. And I already told you in the beginning that this column is just for the new columns or let's say new schema if it evolves. So, obviously, we do not need to carry it forward in the enriched layer. So, actually, you do not need to worry because this column will not, will not, will not disturb your schema. Because that's why it starts from like underscore. And it does not disturb anything. It, it will not disturb any, you can say, your schema. And you can actually see the schema as well. And how you can just see the schema in PySpark? You can simply say `df.printSchema()`. And you will see the schema of your table. And you will simply see like `rescueData`. And there's no big deal in this one. Perfect. Perfect. Perfect.

So, actually, if you just try to rename it, and you can simply say `df.withColumnRenamed()`. And how you can just rename your column in PySpark? We have a function called `withColumnRenamed`. We will simply say `df.withColumnRenamed()`. And you can simply type the column name here. And then whatever new name you want to give it. Let's say `rescueData` without any kind of underscore. Okay? And you will see that you will see that it will be renamed. Rescued data. So, now let's say if you just want to transform it, if you have some values, you can actually play with that with with it. And you can, it's totally up to you. But in our case, we actually do not need this column. So, you can even drop this column as well if you want. So, how you can just drop the column in PySpark? You can simply use something called as `df = df.drop()`. And then you can simply say `rescueData`. And just for your kind information, we didn't save this into the `df`. Let's save it first. Oh, wait, wait, wait, wait, wait. We just saved it with `.display()`. Okay? So, we would need to re-read this. And not a big deal. So, now, just perform this. Okay? Now, just say `df`. Now, you will see that this column is gone. And we just keep this column just till the bronze layer. But after that, it totally depends if we want that data which is coming into the `rescueData` column. We can keep it. Otherwise, we can simply drop that column. Simple. Simple.

Now, as you can see that our data is looking perfect right now. Let's say `orderID`, `customerID`, `productID`, and everything like `orderDate` and everything, right? Yes. `quantity`, obviously, it is in the integer form. `totalAmount`, it is in the float amount. For, like, you can say, float, float data type. So, now, let me just show you some date functions that you can perform. Let's say you want to convert your `orderDate` into any other format, or let's say into timestamp. How you can just do that? Basically, date transformations in PySpark. How you can just do that? You can simply say `df = df.withColumn()`. So, now `withColumn()` is a function that we use to either modify a column or create a new one. It's totally up to us. Makes sense. Okay? So, now `df.withColumn()`. And now what we will do? We'll simply write the column name. Now, here's the thing. If you just mention the column name here, then it will check if this column exists in our table. If yes, then it will simply modify it. Otherwise, it will simply create a new one. Sorted life. Very good. Simply say `orderDate`. Okay? And then now we need to just define the transformation that we need to apply on this particular column. And I want to apply, let's say, `to_timestamp()`. And I will simply use `column` object. And then `orderDate`. Perfect. And now let's display the data as well. Perfect. So, now you will see that our date column is now converted into a timestamp column. And now, obviously, it will be `00:00:00`. But still, now you know how you can just convert that particular, uh, column into a timestamp. Okay?

Now, you will say, "Can you just show us an example of creating a new column?" Because we are really interested in that as well. Yes, bro. Yes. Why not? And here, bro, means you as well. Miss, Mrs., whatever you are. Okay? So, now, now, now, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no,

three functions which is this one and this one rank function and this one. So now let's say I have DF. Okay, I have DF. First of all, let me just create a new DF. Let's say code and I will create a DF new equals DF, obviously. Okay, let's do this because I do not want to make a mess of my DF original DF. Okay, so let's say in obviously in DF new, we would have everything that we have in DF. So let me just display the DF. df new. DF new. Let me just do that. So in this, we have all the things that we have in DF. So let's say I want to create, um, dense rank column. Let's say this is my new DF and I want to perform window function on this particular data frame. Instead of writing the whole function again, what I can do, it is very simple. I will simply create an object of my class which is called windows. Okay, so this is my object. Now, in order to call the function, I will simply say, first of all, let me create the object. Windows is not defined. Are you sure, bro? Didn't we run this? Okay, let me just run this. I thought we we ran this. Now, simply run this. Perfect. Now I will simply call obj dot dens rank and I will simply pass df new. Perfect. And you will see it will run fine. See, it has ran successfully. Now, in order to see the results, I can simply save this in a data frame. Obviously, I will simply say DF result. Okay, make sense. DF results, or you can even save it in DF new as well, bro. Not a big deal. So it is just for your understanding. So you will see that in DF result, I will have that new column ready for me, which is flag 1, 1, 1, 2, 2, and so on, so on, right? How was it? So now, this way, if you have multiple DF, if you're just writing like, let's say thousands of lines of code, you can simply reuse your class. Simple. You can reuse your class. You can simply call that particular object and you can simply pass your data frame and that's it. That's it. Makes sense. Makes sense. Very good. So obviously, we are not going to save these columns, but I wanted to show you how you can just transform your data based on the requirements. Makes sense. Makes sense. Very good.

So now what we need to do, we will simply write our DF which is just with the year column because yes, we want year column but not this black column. So we will simply say data writing. Okay, I'll simply say text and then H3 and then data writing. Perfect. I will simply say DF.right.format. Okay, and this time I will save my data in delta format because in the silver layer, we should save our data in delta format. Yes, you should. Perfect. Mode is overwrite. That's fine. Actually, I can use append as well. Not a big deal. Okay, then let me just give the location. ABFSS silver at the rate, um, what was the location name? Databricks ETFs.core.windows.net and then I will simply say orders. Makes sense. Perfect. Let me just write this and our first table is written in the enriched layer, which is also called as silver layer. Perfect. Perfect. Perfect. Perfect. So this was our extra touch of classes that is called OOP concepts because you should know these things. Okay, now obviously classes are something different like you should know classes and all those OOP concepts if you're just becoming a data engineer and you can also learn this along with this project as well. So it's not a big deal.

So now let's create a new notebook. Okay, I will simply go to workspace and I'll simply say create and I will create a new notebook. It's called silver customers. Silver customers. Makes sense. Let's connect this notebook to this cluster. Okay, so now again, same thing. We will first of all import the libraries. From pispark.sql SQL dot uh functions import ax. Then from pispark.sql.types import ax. Perfect. Now let's perform the data reading first of data reading. Okay, perfect. Now DF equals spark.read.format pocket and then simply load silver at the rate, hm, silver bronze bronze. Perfect. Let's see what do we have inside this. Obviously, we have 2,000 customers inside this and again, we do have one more column. So first of all, without link, let's first of all drop this column because we do not need this need this in our enrich layer. Okay, so I will simply say DF equals DF dot drop rescue data and let's say DF dot display. Perfect. It is dropped.

So now what we need to do? So now, first transformation that I'm going to perform and it is really, really important. You should know about this and it is regarding, let me just tell you the requirement first of all. We have email IDs, right? Do not send emails to these emails, bro. So Anita 65, I know you like you will be sending email. No, these are like pseudo emails. So so now, now no, now no, now no. What we need to do basically in the email column, we have so many email elame email IDs. I want to create a new column in which I want to store all the domains. So that I would know what are all the domains that my customers are using. So that I can effectively provide some kind of, you can say, vouchers or those kinds of uh promo codes according to the domain names. Makes sense. Because there would be some domain names of some organizations, there would be some public domain names like Gmail, email, Yahoo, and so on. So I want to know like what are all the domain names and how we can just do this. In order to perform this activity, what we need to do, we simply need to split our email column. Okay, virtually, virtually, not physically. Then we need to pick only the value after add the rate. Wo, so we have to perform multiple things for this transformation. That's why I picked this transformation so that you can learn PySpark. So what's the approach? First of all, I will split the function like split the column. Then then I will perform array indexing to just grab the last value of the list because whenever we just perform split operation, it creates a list of all the values based on a delimiter or based on a separator. Makes sense. Makes sense. And what is the separator in our case? Add the rate. Very good. Let's do this, then you will understand better.

So now let's say no, no, no, no. Now let's say DF equals DF dot with column. Obviously, because we are creating a new column. Okay, now let's say I want to create a column called domains. I will simply say split column email. Okay, and what should be my separator? It's at the rate. Done. Then after performing this, I want to grab the one array, one element of the array. One means like next zero is the first one, one is the next one. I can also use minus one because it is the last, but we just have two, um, items. So it is better to use one. And that's it. I can simply say DF do oops. What's wrong? Uh, wait. Dot column. Oh, I see there's a typo. Uh, I just need to enclose it and then and then column. Okay, this is fine. Then here, uh, wait. Column is this, then this, then this. Oh, it's fine. Okay. Column. I hate parentheses. Okay, this parenthesis is closing here. Then split column is here. Then width column is here. Oh, that's it. We are good. We're good. Let's do this. Perfect. So now you can see I have successfully fetched all the domains. Fun, like first is called rion.org or kelly.com and so many other domains, right? See so many other domains as well. So many domains. So many, so many, so many domains. Obviously, there will be like so many, uh, repeated domains as well, but yeah, I can just see domains as well. Makes sense. Yes.

Now I want to do something special. What's that? Okay, so now what I want to do, I know there are some popular domains. One is obviously gmail.com. Com. Second one is yahoo.com. Okay, and the third one is, uh, let's say king.com. Let's say let's say it is the third one which is very popular. Okay, and we can actually find this. Okay, okay, okay, okay. Let's perform one transformation first, then we will just jump on to the special one. Okay, now let's say, uh, I want to know like how many users or how many customers do I have for gmail.com or how many customers do I have for yahoo.com and so on. So how I can just do that? I will simply perform aggregation functions. So in order to perform aggregation function, I'll simply say, uh, DF dot group by and I want to perform group by on domains column. I will simply say ag, that means what kind of aggregation I want to perform. I want to perform count. Okay, count of let's say customer ID because customer ID column is the unique ID here. Customer ID. That's it. And then if I just want to provide a good name to it, I can simply use the dot alias and I'll simply say dot total, um, customers. Makes sense. Perfect. Now let's say dot display. Perfect. Perfect. Perfect. So wow, we have got all the things and now let's actually sort it as well because there are total 811 domains. So I will simply say dot sort on total customers and ascending equals to false. Perfect. So this is a transformation in which we have performed so many transformations. See, see, see, so many, but that's how you learn. That's how you just combine all the transformations and that's why this this is the like you can say one of the major reasons of building projects where you just apply all the knowledge that you gained so far. Let's say you already are aware of all the functions in PySpark, but while you build the projects, you apply all those functions and you actually combine so many things together, right? Very good.

So now I got my top three domains, Gmail, Hotmail, and Yahoo.com. Very good. Now I want to perform something special. What's that? Why are you smiling, anala? So the thing is, I want to apply a filter on top of these three domains. That means, let's say I want to perform a filter on DFGmail and I want to just filter all the customers of the Gmail domain. I will simply say DF dot filter, then I will define the condition. I will simply say column of, um, domains. Okay, and then equals to equals to gmail, gmail.com, sorry, gmail.com. Yeah, perfect. And I can simply say display. So this way, I will only see the gmail.com. Makes sense. Now let's say, now let's say I want to do the same stuff. I want to do the, let me just do this. Now let's say I want to do the same stuff for, uh, for Yahoo. Okay, and here yahoo.com. Okay, and then Hotmail, hotmail, hotmail. Perfect. Makes sense. Yes. Now let me just run this and you will see what I'm talking. And it has just run three things. Okay, one by one. Makes sense. Because obviously, first this code will be running, and then this code will be running, and then this code will be running, like one by one, one by one, one by one. And just to prove this, I can simply import time module. Okay, I can simply say import time. Okay, so I can simply say time dot sleep 5 seconds. Okay, then time dot sleep 5 seconds. Then time dot sleep 5 seconds. Let me just rerun this, then you will see that first it will run and wait for 5 seconds. Okay, so it is now waiting for 5 seconds. Perfect. And now perfect. So this way you can actually perform some kind of time modules if you just want, let's say, um, any kind of delay if you just want between your code or any any anything. So for now, in this particular thing, all these code like DF hotmail, DF Yahoo, DFG Gmail, like actually processed together and it returned all the result together. Okay, makes sense. Makes sense. Makes sense. Makes sense. So now this this was like filtering of the data frame based on the domain. Makes sense.

Okay, now let me first of all remove this, um, cell. Why? Because it is taking so much of space. I will keep it here, but I will not run it because we know the output. Okay, just to have some space. So let's say now our DF is looking like this. Okay, now we have already seen like how we can just apply split, group by, and everything. Now let's say that I want to perform a kind of new column and I want to create a full name instead of last name. How we can just do that? Basically, we can do that using a concat function. So basically, we have some text functions as well, and let me just create that. I will simply say DF equals DF dot width column and I will simply say full name. Okay, and within this full name, I will simply say concat. See, I think it is hearing my voice. I I am so, so, so sure. Oh man, oh, this was in my mind because obviously when we have created a full name column, we do not need first name and last name column, so we will simply drop it. But it is so smart. But I will simply say TF here. So let's create and let's see what is the result because we do not want like two different columns, we just want one column and that's it. This is full name. Perfect. So this is our data that is looking like this and we are pretty much satisfied with this particular transformation or you can say set of transformation that we have applied on this particular table. Makes sense. Okay, now let's write this data. TF dot write dot mode dot format. Okay, yeah, perfect. Now let's run this and I want to just save it in delta format and I want to do append. Okay, makes sense. Perfect. Now let's do this. Perfect. Perfect. Perfect. So now it has written the data. I will confirm as well. I will simply go here in the containers and silver and yeah, we do have customers table as well and we do have two partitions and in this and the silver, sorry, in orders, we do have two partitions there. Yeah, that's it. Good. And we can also confirm it just to read or we can just perform the reading at the end of this particular, you can say, silver layer, um, part, so that we can confirm. Okay, all the data is looking fine. Okay. Perfect.

Now let's create another notebook for our third table, which is silver products. Perfect. Let's attach the notebook with the cluster. Okay, perfect. First of all, importing the libraries. That's it. Now let's pi sap by spark group pi spark. Okay, now let's write data reading. Perfect. Now what we will do? Perfect. Let's see this data. What's the typo, bro? What's the typo? What's the typo? Databricks ed windows windows windows products. Hm, oh, date. Databricks. Perfect. Now let's see how this data look like. And this particular, obviously, we first need to remove this particular column. So let's do that. TF dot drop data. Perfect. Okay, now let's run this command one more time and we should not see this data. Perfect. So now, obviously, we have these things. Product ID, product name, category, brand, and price. Very, very, very good.

Now I'm going to tell you something very, very, very special, which is really important for you as well. It is called functions. What are functions, bro? So basically, we know that we create something called as user defined functions in PySpark, but those functions will stay there till your program ends. But we can actually store our functions in our catalog so that we can reuse it. The same way we do it in SQL. Let me just show you. If you just click on this catalog, okay, and then click on this ribbon to have more space, database catalog, within this, we have bronze schema. Within this, you can create something called as function. Really? Yes. And the good thing is, the syntax is almost, almost similar to the TSQL that we perform in MSSQL server, and you can reuse that function. So let's say you are creating a function and you want to use that function in another notebook, or let's say you want to use that function in all the other notebooks, then obviously you cannot rely on UDF, right? Because UDF will not be there in all these sessions. But this function can be reused and it will be stored here. And I'll just show you how you can just create function to reuse it. Makes sense. Obviously, we'll be just creating some basic functions so that you will understand the concept first, and you can actually scale that particular solution to such a great height, and it will be really, really, really helpful for you. Okay, so let me, basically, we have two functions, uh, scalar and table function, which return tables. So let me just first show you the scalar function. How you can just create scalar function. Let me just show you.

So now let's have a look on the functions. Okay, so as we know that we have this brown schema. Let's create the functions that will be retained throughout the session and that can be reused even after the session gets killed, bro. Because the thing is, you want to create some function. If you have used SQL, you know that we create functions so that we can reuse it, right? These functions can be long, these functions can be really, really complex. So obviously, if you just want to write the code, you will simply write the code one time and you will just retain it. Okay, so that's why we have functions and the Unity catalog. We can actually create functions. Let's create that. Okay, let me just create a heading H3 bold and functions. Makes sense. Makes sense. Okay, so now basically, we can use SQL and Python both. So first of all, I will just show you SQL and then Python. Okay, and I will also show you how you can just call those function using SQL and Python both. Makes sense. Makes sense. Okay, very good.

So in order to do that, uh, we know that in our particular table, this is the one and we know that we do not have any kind of table. So let's create a temporary view first of all, so that we can just test our values. So how you can create a temporary view using DF? It is very simple. You can simply, um, create something called as DF dot create or replace temp view. Simple. Uh, this is a products view. Perfect. Let's create that. Yeah, DF is not defined. That makes sense. Why, why, why, why, why? Because because because my cluster was dominated. So I will simply say run all. Perfect. Okay, perfect. So our view is created. Now, this view is like just a temporary view that we can just use it because let's say our function is created. Now, obviously, if you just want to use that function using SQL, you should have some SQL objects, right? Obviously. And that's not even a big deal because we have regions, but in regions, we do not have anything that we can just apply. Okay, so what we will do, we will simply create a function and I will simply say create or replace function and we need to create a function in Databricks kata dot bronze schema, then dot function name. So function name, I will simply say, let's see what do we have here. Um, okay, we have something called as price. So I will simply say, um, price after tax. Makes sense. Let's, let's cut tax or, um, yeah, or let's say discount. Discount is like 10% or like price after discount. So discount is 10%. So we will simply provide the 90% of the amount to that particular column. Makes sense. Okay, very good. So I'll simply say function name as discount, discount funk, just to make it more readable. Then we simply need to pass a parameter because obviously we will be just importing the parameter. So I'll simply say P and then price. Makes sense. And what is the, um, data type of this? It is double. Okay, then you need to say returns. What it will return? It will return, uh, it will return something with data type called double. So this is our scalar function because we'll be returning only one value. It can be string, it can be double, it can be float, it can be integer, but it will be just one value. That is why it is a scalar function. Makes sense. Now we need to say which language we are using. We are using SQL. Makes sense. Very good. Now it's time to actually define what it will return. It will return, uh, P price into 0.90. Makes sense. That means just the 90% of it. Let's create this and let's see if it is done. Yes.

Now I want to perform this function using SQL on top of this view. So I will simply say select, um, product ID. Now, if you just want to perform any kind of function in SQL, what you do? You simply write the function name. So in our case, it is called discount. Okay, funk. And then from, uh, products. Makes sense. Let's run this and you will see, uh, discount function search path. Uh, uh, uh, wait. Is there a typo? Is there a typo? Cannot resolve on search path. Okay, uh, okay. Oh, we need to pass the catalog as well. So catalog is Databricks kata dot, um, schema name. So schema name was bronze, because only then it can just identify, hey, this is the column, like this is the function that we are just performing. And perfect. So as you can see, 1681. And what was the only original price? Actually, I can just display that as well, just for you. Perfect. So this was the original column, and this is the discounted price that is just the 90%. That means 10. So my camera was turned off. So if you didn't see my face, so now you can see. So I was saying that this is the price, okay, and this is the discounted price, that means price after applying 10% discount. So this way I can use this function again and again in all the, all the, all the objects stored within this schema. Actually, this is a catalog. So click on this, then bronze, then see. Now I have two, two options. Tables and functions. Early it was only tables. Okay, click on functions. So this is the function that I can reuse every time. So this is the way to use it in SQL. Let's say you will say an Lamba, why are we creating temporary views? We want to just use it in the, um, data frame, uh, directly. Okay, you can use it. You can simply say DF equals DF dot with column. Okay, let's say you want to create a new column and you will simply say discounted price. Okay, then you need to use something called as expr, because whenever you want to use SQL functions or functions in a SQL way, you just use expr and then within the expr, you just write all the function in the SQL way. So I'll simply say, um, Databricks, ka dot bronze dot, uh, function discount function, and then I want to apply it on the top of price function. Same way that you use it in SQL, same way. Then we use expr. Okay, and that's it. I can simply say DF dot display. Perfect. So as you can see that we have an error. Wow, wow, wow. What is it saying, man? Oh, just a typo. I forgot to add S and that's it. That's it. That's it. That's it. So perfect. I can see this is applied on my DF as well, directly. Just remember, you need to use exprum in the SQL way. That's it. This is the, you can say, flexibility that that we have within the data frames. Very good.

Now Lamba, can we just build the same function using Python as well? Yes, you can just do that. So basically, or let's create a new function because this is a very simple function. Let's create something. Let's say I just want to make upper, lower, something like that using Python. Okay, so I will simply say, and let's do this transformation on top of category or let's say brand. I want to make a brand column as upper. Obviously, I have upper function in PySpark. I'm just telling you how you can just create the functions using Python and using SQL because in the real world, you'll be creating complex functions and you would need to save those functions. Okay, so I'm just making your fundamentals really, really, really strong, and this is a new thing. So you should be aware of this thing. Makes sense. This is a new thing. Yes, this not before in Databricks. Okay, so how we can just create a function using Python? So the 50% of the code is same. Okay, like the statement or definition code is same. Create or replace and function, then Databricks kata dot bronze dot, let's say a function, uh, uh, upper upper funk. Makes sense. Then obviously, I will take P parameter, brand, and this is of string type, and I will return string. Okay, and this time I will simply say language Python, and this time it is different. We need to use as, then dollar dollar. Where is dollar? Where's dollar? Here it is. Dollar dollar. And then whatever I I'm writing in this particular area, it will simply return that. So let's say I want to, it is hearing my voice. I'm telling you, bro. I'm telling you. Why didn't it pick lower? Why I just said that I want to make this function as upper? It picked upper. I'm telling you, this is an AI error. Oh man. So as we know that we simply need to write return. And even if you just write, let's say, any anything, um, anything, let's say you want to just write for loop, you can just write for loop like for I in your P brand. Makes sense, right? And then you can simply say, um, um, if I do is upper, uh, yeah, you can say that, or you can simply say, um, return I do return. Simple, simple, simple. So what it will do? It will simply return that one by one, by one, by one. That's it. You can do anything, anything means like you can just write even I would say your UDF function in Python as well. You can simply say DF or you can just write anything in this particular area. Makes sense. So for now, what I'll simply say, return P brand upper, which is the common thing that we do in Python. Or let's say, if I just want to reverse a string, I can simply say that return P brand and P brand and then list and then I can simply pass. Are you getting my point? My emotions, like what I'm trying to do right now? You can write anything and you can just save it. Makes sense. Makes sense. Makes sense. You can literally do any, any anything. So let's, okay, just perform this for now. And now if you want to just do the same thing, you can actually use the same function here. Let's say select a strings from, um, whatever your function name is, uh, first of all, let's do product ID, then brand. Perfect. Let's see the output of this, and you will see that that particular function will be applied on top of that particular column. Makes sense. Okay, and it has applied the function successfully as you can see. All the values are turned into the upper case. Again, I'm repeating, you should only use this function if you just want to create a complex functions that cannot be achieved using PySpark and you want to save it. You can just do it. And obviously, you can govern it as well using Unity Get a log. Makes sense.

Now let's actually write our DF. DF dot write dot format. Format will be delta. You already know that. Then dot mode will be append. Then dot save or dot option path will be oops. Perfect. And then dot save. Makes sense. Let's run this and our third notebook is also done with some new learnings that what we can do. Makes sense. Makes sense. Makes sense. Very good. So now let's quickly create our fourth notebook as well. By the way, fourth notebook is like very, very, very basic notebook. Let me just show you. Let me just show you. Go to reasons or go to workspace. Go here. Click on new notebook. And I will simply say silver regions. But still, you will learn something new in this as well. What's that? Let me just show you. What you will learn in this? Basically, we need to read the data and the data is in the form of table, as we know regions table. So what we need to do? And this is a data table. So how you can just read the data table? Let me just show you. You will simply say DF equals spark.read.table. That's it. You do not need to worry about any location or anything because this is a table. You will simply say, um, Databricks ka dot bronze dot regions. And that's it. That's it. You can simply say DF do. And this data is just about regions. Four, four regions. And obviously, we need to remove this particular column. Let's remove that. DF dot drop rescue data. And that's it. Because everything else is fine and we are satisfied with the with the with the with the with the with the transformations. By the way, let me just tell you, this is a special requirement of the project where I want to just show you this particular file will not be the part of, um, star schema, but still it will be there. Why? Because this is a mapping file that you want to travel it through medallion architecture and you want to serve this file to the stakeholders. So this is not the part of star schema, but still this file should be there. That is why it is just a static file and it is just a, you can say, mapping file that we are creating and we will simply create a kind of object in the gold. That's it. Makes sense. Very good.

So let's write this data. TF.t write.format delta and then dot mode override. Yeah, it can be override because this is just a small file and static file. It can be over over overwritten every time. Okay, and then obviously, do not save it as a table, just save in the location. So there is this is another way to write your code in external location instead of using option path, you can directly do this as well. And this should go to silver. Perfect. So now it's time to query all the data that we have, just to make sure. Okay, what's the error? What's the error, bro? Hm, silver Databricks. Oh, Databricks ET. By the way, it suggested me. Okay, okay. Perfect. Let's now quickly read all the data. Let's say DF1 equals or let's say DF equals spark.read.format delta and then dot load. Let's read this one first. TF.dod display. Because obviously, before creating the gold layer, we should be sure that okay, our data is looking fine. Okay, this is fine. Now, next is orders. Let's see that. Okay, this is fine. Uh, regions. Oh, regions is done. Customers. Uh, 2,000 customers. Yes. And then at the end, products. Okay, perfect. Perfect. Perfect. Perfect. So all the data is fine. Now it's time to actually get started with our gold layer. And let me just tell you, gold layer is very, very, very, very, very, very special because we going to create star schema, we going to create dimension tables, we going to create fact tables, we going to create slowly changing dimension type one, we going to create slowly changing dimension type two. So there is a lot of, a lot of, a lot of learnings. Okay, let's see what do we have in the gold layer.

So bro, now, now, now, now is the most important, most important, most important part of the video. I'm not lying. Obviously, all the parts are important, but this is the most important part because in this particular part, we are going to create our gold layer, dimensional data model, star schema, slowly changing dimensions, and a lot, lot, lot of things. So basically, I will first show you because obviously we have one fact table in two dimensions, and one dimension is customers, second dimension is products. So we will be creating our customers dimension as slowly changing dimension type one, and, uh, products table as slowly changing dimension type two. So that you will have understanding of both the most popular dimensions because type three is like rarely used. Type one and type two will be used. Just be with me and let me just give you a disclaimer. Um, this is a complex area. Okay, so you may need to rewatch this part again and again. It's fine. It's fine. Because obviously, whenever we are just learning a new things, new thing, it's not like you will be just grabbing all the things in first go. It's fine. It's fine. Okay, so just be happy and just be with me, and you will be happy. Okay, so now without wasting any time, let's create our notebook. Go to workspace. Click on create notebook. Perfect. I will simply say gold and then I will simply say gold customers. Makes sense. Very good. Because there will be like so many things which will be new to you, and trust me, it's really, really, really important. Okay, and now we would first need to create our schema as well. Obviously, because we'll be storing our column, storing our tables, uh, so let's create our, um, let's create our schema. So you can simply drop, click on this dropdown, click on this three dots, uh, or you can simply go to this menu and catalog and then you can go to this catalog. Basically, you can simply write create schema schema name. That's not a big deal, but I just want to show you through the UI. It's better to see and create. Click on this catalog and then click on create schema and let's create our gold schema. Okay, create. That's it. Perfect. Perfect. Perfect. Let's go back to our notebook. What do you want, bro? Oops. Recent gold customer. Yeah, perfect. Let's collapse it. Perfect. So this is our cluster. Okay, so don't worry, I will just attach this notebook notebook as well. So just, just, just be with me. First of all, I want to create a flag in the beginning of the notebook. Yes. Why? Because in this particular notebook, I'm going to create a real world solution for SCD type 1 dim, and in the real world, we perform something called as initial load/full load plus incremental load within one notebook. So you need to prepare your notebook in such a way that it can handle the dimension table creation for the first run and for the incremental run as well. So just to give you an overview, so obviously, let's say you have a dimension table, obviously, and if this table is being created for the first time, that means this table is not there, then obviously you will simply dump this table. You will simply write this table. But once this table is created, you cannot append the data. No, you need to apply something called as upsert, which is update plus insert. That means you will only insert the new records and update the existing records. And obviously, there is a rule. There is a rule that we always create a kind of dimension surrogate key. So you need to take care of that as well. Everything in one notebook. Again, that's why it is a real world project. It's a real world project, right? Obviously. So just be with me.

So first of all, I will simply read the data because our source is silver data. Makes sense. Yes. Very good. So now, in order to work with silver data, like we could have created the tables as well on top of it, that's not a big deal because we can simply use spark.sql and that's it. Ideally, we should have created tables on top of those silver, you can say data, but we just have the, uh, you can say delta, uh, format data. That's it. But ideally, we should have created those table. But it is fine. But even if you want to create those tables right now, you can create it. Let me just go back to the recents notebook and let me just open silver customers or let's create. Yeah, silver customers. That's it. Like it's up to you. It's it's a good practice to just create that. So just create table on top of it. And how we can just create a table? We will simply say create table. Okay, if not exist. And this table will be, uh, Databricks kata dot silver. And we do not have silver, uh, schema. So let's create silver schema quickly. And I will simply use a code cell and I'll simply say create schema Databricks kata dot silver. Done. D. Yeah, perfect. Now it is there. Yes, I can check. Uh, let me refresh. Uh, yeah, here it is. Silver. Perfect. So I will simply say customer ID and then email. And what do we have? By the way, we do not even need to define the schema because in the delta log, we have everything there. We can simply say, uh, using delta and then location. And location will be abs, um, silver and then at the Databricks customers. That's it. This is the location. No parent location was defined. Oh man, it gives me wrong suggestions. It's Databricks ET. It autofills for me and then it gives me error. Databricks. Okay, perfect. So this is the table created for me. Now I can simply query this data like this. Select a from Databickscattera do.sc customer. Perfect. So it will just give me this particular table. Oh, it has created with customers silver. No, bro. I will create it with customers. Okay, let's create with customer silver. Not a big deal. Not a big deal. Not a big deal. Not a big deal. Not a big deal. And perfect. As you can see, we can just see the data as it is. And even if we just rerun this code, it should run fine because we have said create table if not exist. Very good. Okay, so this is fine. This is fine. This is fine. Let me just apply this. Let me just add the cable. Otherwise, you will miss my face because it just, uh, turns my screen into, I think, black screen and then, okay, so this is done. And let's do the same thing with the others as well. Like with the, you can say, yeah, let's perform this particular thing with our other tables, which are these ones. Where it is, where it is, where it is, where it is, where it is. Okay, just go to recents and products. This time, yeah, silver products. Makes sense. Makes sense. Makes sense. Makes sense. And this time it will be product silver, product silver, and this is products. Very good. So this is also created. Now let's create for the orders table as well. Okay, now let's say orders. Uh, there it is. Here it is. Here it is. Yeah, here it is. Okay, let's do it here. Customers, orders, customers, orders, order silver, and then orders. Perfect. Now what I will do, I will simply create another one. That is the last one, which is regions. And regions, uh, yeah, regions. Okay, region silver, regions. Perfect. Our tables are done. Now let's quickly move back to the gold layer. Okay, gold customers. Perfect.

So now what I need to do, I simply need to first read my data, that is my source, which is silver customers. Makes sense. Okay, very good. And obviously, I just mentioned that we need to first create the flag. So let's create a flag first of all. I will click on edit and then I will simply say add parameter. And then I will simply name it as, let's say, I want to name it as, click on settings and I will name it as, uh, init flag or not init flag, um, let's say load flag. Is it initial load or incremental load? Basically, load flag. Makes sense. And, uh, initially, it is one, because we do not have any table in our gold layer, so it is our initial load. And obviously, when we'll be performing the incremental load, we can simply change the value from here. Makes sense. And in the production as well, what we do in the real world, we first load the data, we first run our pipelines with this parameter, then once it is deployed, we just change the value of this to zero, and zero will be there for rest of the runs for rest of the runs. That's it. Makes sense. This is the way to work in production as well. So now let's do one thing. Now let's read our data. Okay, so I'll simply say data reading. Data reading. Now is the thing. So now, just so our process has started for initial and incremental. Why so? Uh, first of all, obviously, I want to read my data as my source. So I will simply say DF equals, uh, spark dot read dot table. Makes sense. Or let's say spark.sql SQL and then I will simply say select a from, um, silver dot, uh, customer silver, right? I will simply say this, right? This is my data. Okay, data reading from source. Oh, silver cannot be found. Are you sure? Are you sure? Oh, we didn't define the catalog. Okay, perfect. Okay, perfect. So now I will simply say data reading from source. Very good. So this is my data and you can simply see this as well. Not a big deal. You will simply say DF do. And you will see the data. Perfect. Perfect. It is fine. Good.

Now what I need to do? As my second step, obviously, I will first of all remove the duplicates based on the primary key. And what is the primary key? Primary key is my customer ID. Makes sense. Obviously, there should not be duplicates, but still, we just need to be double sure. So we simply remove some duplicates. So I'll simply say removing duplicates. Okay, or simply say removing duplicates. Uh, removing duplicates. And now simply say DF equals DF dot drop duplicates. We have this function. And then we can simply say subset equals to, because we simply need to remove the duplicates from the column called customer ID. That's it. Okay, so I'll simply say customer ID. Okay, now simply say display DF.tl limit, because I just want to see 10 records. Okay, perfect. Makes sense. Makes sense. Okay, and let's remove this. Perfect. So now we have removed the duplicates. Okay, now what we need to do? We need to assign a surrogate key column to our data frame. Makes sense. Yes, to some extent. Yes. What is a surrogate key? So let me just show you the data. So surrogate key is just the pseudo key that we create in our dimension. So that we can easily apply joints. For now, what is the primary key? Customer ID. So what we do? Whatever is our primary key, we simply create something called as dimension surrogate key. Okay, dim key. It is called dim key, dimension key. So we simply create dimension key on top of that column. So that we can efficiently apply joins. Instead of applying on this column, we can simply apply joins on that particular column, which is called dimension key. Dimension key can be created with the help of so many functions. We can easily use row number, because it will be a unique identifier of the record. That's it. 1, 2, 3, 4, 5, 6. And we have already a function called monotonically increasing ID in Spark. We can also use that. So I will be using that function. It is really good. So I will simply import it from pispark.sql.functions import ax. From pispark.sql.types import perfect. I'll simply say surrogate key. Surrogate key. All the values, you will get to know what is this, because we are just providing surrogate key on all the values. Okay, okay. Makes sense. I will simply say, uh, DF equals DF dot with column. And I'll simply say dim, uh, customer key. Okay, and I'll simply apply monotonically increasing IDC. This one. And I will simply add plus one. Why? Because it starts from zero. So I can simply say plus one. Makes sense. Uh, makes sense. Yeah, so we cannot apply plus one like this. We need to simply say lit, because it does not take constant like this. We have to wrap it in the lit. Okay, because this is a constant. That's why. So let's run this and let me just show you the output of this, like how it will look like. DF dot with column. Oh, sorry. DF dot display. Uh, DF dol limit. Perfect. So this is my dim key. See, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Perfect. Makes sense.

Now is the real working started. Now, just tell me one thing. If this is an initial load, that means we do not have any kind of table.

We can simply write this table in the gold layer. Makes sense. But, but, but, let's say in the next run, we have new data. Let's say we have 50 more records. Makes sense. If you will be creating a surrogate key on that particular column from one, does it make any sense? Obviously not. Why? You will say, "Anal Lamba, because we already have 2,000 customers." Okay, so the next surrogate key should start from 2001. Exactly, exactly. That's what I wanted to hear. Exactly. So now, what is our task? Our task is, whatever data frame that we have, whatever data frame we have, we need to divide that data frame into two parts: one is new records, second is old records. Why? Because in the old records, we do not need to create any surrogate key because those records are already there. And in the new records, we need to create a surrogate key. But, but, we need to start from the max value of the surrogate key stored in this particular table, which will be in the gold layer. Makes sense. Your data frame will be divided into two parts: one is old records. Keep it aside because we, like, we do not need to add any surrogate key because those records are already there. Second thing is for the new records, we need to create a surrogate key. Plus, plus, we need to say that the surrogate key should start from the max value of the previously loaded table. If we have 2,000 records, the surrogate key should start from 2001. If we have 10,000 records, the surrogate key should start from 10,001. Okay, and I know now you have got the concept. Now, let's see how we can implement this. Okay, let's see. So now, what we will do, we will simply go back to our surrogate key step because we do not need to create surrogate key for all the values. First, you will filter out what are the new records, what are the old records. Makes sense. Very good. And just to give you a kind of understanding, in our end table that we create in our gold layer, the end dimension table, we create two more columns: one is create date, one is update date. Obviously, because we need to keep a track when this record was created, when this record was updated. Makes sense. Yes, it makes sense. So, why this information was required? Because you will be just dealing with this right now. And let me just show you how. So, the thing is, let's first of all decrease some brightness. It is, it's 7:53 p.m. here. So, obviously, I started in the morning. So, basically, before applying surrogate key step, we need to first of all filter out previous and old records. Makes sense. Okay, let's do that first of all. Okay, we'll simply say filter or let's say, yeah, or let's say dividing, dividing new versus old records. Makes sense. Makes sense. Dividing new versus old records. What did I click here? Uh, okay. New versus old records. So, now, how we can just do that? We will simply, so what, what, what is the solution? Let's say I am creating this data frame. This DF is the new data frame. This one. Okay. Now, if I want to see what records are there and what records are not there, how we can just do that? You will simply say, "An, you can simply create a data frame on top of the gold table and just apply a join." And if we see any nulls, that means those are new records. Yes. But what if we do not have any table? Because this is the initial load. So, how we will just do that? So, in this particular scenario, we know that all the records are new. But still, we want to programmatically manage it. And what will be the scenario in this case? We will create a pseudo table. What is a pseudo table? Let me just show you. Pseudo table is, we will simply grab the main, main columns. That's it. And what is a main column? Obviously, the joining key plus your DSS create date and DSS update date. Makes sense. Or basically, just the surrogate key. Because what we are trying to do, we are trying to figure out what are the records that are already there. Makes sense. Okay, so I will simply create an if condition. I will simply say, "If init flag, init load flag, you will now understand everything." Init load flag equals to equals to 1, or let's say equals to equals to 0. That means this, this is very simple. Equals to 0. Very simple. Because we know that we have table created in the gold layer, that's why it is an incremental load. Okay, so what we will do, we will simply say DF old equals to, uh, spark.dosql. Okay. And then select ax from datab bricks.gold.let's say the table name is dim customers. Let's say I know this is not existing right now, but let's say. So, we will simply say select a from that. That's it. Because we want to simply grab it. Okay. Or, um, let's say we just want main columns. What are the main, main columns? Because we are just, just, just want to, we are just trying to get the columns such as, you can say, uh, first of all, surrogate key, obviously, because we need to just see like, okay, what was surrogate key or anything, just to see anything. So, we'll simply say dim customer key. I know this is critical, this is complex, but try to understand. Wait, you will understand everything. Just, just hold on. Dim customers key. Okay. Then, what you will grab? You will grab the joining key. What is the joining key? Customer ID. Okay. By the way, this column is not there because this is this table is not created for now, right? But we are assuming that this table is already created because we are creating, or you can say managing the notebook programmatically for both these scenarios. So, you need to imagine, right? So, customer ID. Then, we'll simply say, uh, create date and update date. Create date and update date from this particular table. And when we just write our code in multiple lines, we simply use three single quotes. That's it. So, this is my DF old. If my table is there, if my table is there, because obviously then it will simply bring all the columns such as dim customer key, dim customer ID, create date, and update date. Makes sense. And else, else means if my table is not created, then I will create a pseudo table so that I can easily apply a join. And every time the join will be returning all the records because there is no table created there. Just look at the code that I'm going to do. Do so, DF old equals spark.dosql from this. And what I will do, where 1 equals to 0. What, what it will return? What it will return? It will only return the columns, just the schema. That's what I want. That's what I want. Okay, so I can simply say create date and update date. Okay, let's run this. Uh, init load flag is not defined. Really? Uh, okay. Makes sense. Because we didn't load this particular thing. So, we will simply say init load flag equals db utils.widgets.get. And it is init load flag. Makes sense. Perfect, perfect. Now we can just use it here. So, now it is done. Uh, what's the issue? Database ka gold dim customers cannot be found. Uh, okay. Makes sense, makes sense. Because obviously, it is not here. So, now it, what it is saying? It is saying, "Hey, this table is not here." So, obviously, if we are just trying to get anything, or let's say any schema, so obviously in that particular scenario, we cannot say, "Hey, you need to just bring this data from, let's say, this particular table," because this table is not there, right? This table is not there. You can simply say, simply run this and you can simply provide from, like, any from, but it should be a decent table. Decent table, like there should be a table. So, you can simply say any table, like bronze, silver, or anything. Silver.gold.or databex ka.silver.customers.customer.silver. Just run this. Uh, dim customer cannot be resolved. Oh, okay. Simply write zero here. Why? Because obviously, these columns are not there. And these are like the pseudo columns. So, you will actually understand what we're trying to do when you will actually see the DF old. Otherwise, you will say, "Hey, what we doing? What you doing? Just be with me. Just be with me." So, simply say zero as dim customers key. Because we know that this is a pseudo column. We are not going to use this DM, this thing, but we are simply creating it. Same thing here, zero. Same thing here, zero. And you can even create zero as customer ID as well. Not a big deal. Because this is, this will only return zero every time. And not even zero, we just need the name. That's it. Trust me, we just need the name. So, I'll simply run this and it is fine. Now, let me just show you DF old.display. And you will see just the schema. That's what I want. See, I just wanted this, like, I just wanted to create something so that I can get the joining key, create date, update date, and obviously dim key. That's it. I do not want anything else. I do not want any value. So, that's why I just put zero. Okay. So, now, is the thing? So, now we have DF old. Now, just listen to me carefully. Now, in our initial load, in our initial load, we have this particular table. I know it is empty, but this is a table. So, now, what we will do, we will simply apply the left join with this particular table on this particular column. And, and because obviously now it is empty, but for the incremental runs, for the, you can say, subsequent runs, there will be some data. You can imagine some data. So, what will be the deciding factor between existing records and new records? This column. Makes sense. Let's say customer ID, uh, customer ID 2011. Okay, has some, uh, obviously it will not have any kind of well, let's say, even have some kind of value. Let's say zero or anything else. Because obviously, this is the empty table. But in the real world, obviously, not in the real world, like in the subsequent runs, in this one, we will be having values, right? Dim customer key, dim customer ID, create date, update date. Obviously, if we have some values here, in any column, in any column, if we see any column, like any value, this column, this column, this column, but I personally prefer this column because this column is easy to match. So, if we have any kind of value here, that means this customer ID is there in the table. Oh, okay. So, if any customer ID is there in the table, that means we have customer key, dim customer key associated to it. So, we do not need to create a new one. Makes sense. We will only create new ones for new customer IDs which are not in this table. Makes sense. Now, let's do it. So, our next step is applying the join. Applying the join. So, I'll simply write applying join with the old records. Okay. Applying the join with the old records. And I will simply expand the size of this so that you would know that this is the subsequent steps. So, we are dividing new versus old records. And these are the substeps that we are doing: applying join and all. Okay. So, now I will simply apply join with my DF. DF and I will call it as, let's say, DF. DF. Makes sense. Or let's say DF join. DF join equals DF.join. And what is the second DF? Let's say, what is the second data frame with which we are applying join? It's called DF old. Makes sense. Then, what is the column? This is the column customer ID. And it is a left join. Okay. Let's see what do we have. So, I will simply, we say DF join.display. And obviously, this is an initial load. So, all the values will be null. All the values will be null. And you can see all the values will be null. See, dim customer key, customer ID, create date, update date. All the column values are null. So, that means all these records are new. That's the truth. Because this is an initial load. All the values are new. That's what we want. That's what we want. Makes sense. Okay, very good. Now, we need to take care of some things. What, as you can see, that we have same column name that we should not have. Because obviously, this will create a kind of confusion between the columns when we will be just filtering out the values. So, we'll simply rename this particular, like, these particular, um, columns. And how we can just rename it? We can simply go back to our DF old. And we can simply say DF old equals DF old. Um, with column renamed. Okay. With column renamed. Or, yeah, with column renamed. Or you can obviously rename at this step as well. So, I will just do it here instead of doing it here. So, I will simply say rename. And let's do it here. Yeah, perfect. Renaming columns of old, of DF old. Makes sense. Okay. I'll simply say DF old equals DF old.with column renamed. And dim customer key will be changed as old dim customer key. I will simply add old as a prefix. Just for the identifier. That's it. Okay. And then I will simply add this same thing with others as well. So, with column renamed, customer ID, and create date, update date. That's it. Very good, very good. Now, let me just run this. And I have just added space. That's why we have an error. And I have added space. So, let's remove some space. Perfect, perfect. Now, let's reapply the join. Just to eliminate any confusion. So, now you can see that we have new join. Oops, we have an error. Oh, yeah. Now, obviously, this is a new column. So, we need to update our join. This is, this will be applied with join called as old, old customer ID. Makes sense. Very good. Now, you will see that confusion is gone. That we know that old dim customer key is the old customer key, not the new one. Or it has a different name. So, it's not a big deal to just, um, differentiate between the new and the old. Makes sense. Very good. Now, bro, it's not easy to create dimensions. Okay. So, it's not an easy thing. So, just feel happy that you're learning and you know how to do this. Okay. And I told you, you need to watch this part, LT, at least three times. At least three times. Trust me, because that's how you learn. That's how you grow. So, now it is fine. Now, let's separate the new, uh, records and old records. So, as I just mentioned, all the old records will be having some value here. And all the new records will be having null. So, let's do that. Let's quickly do that. So, now we will be simply saying that separating, separating new versus old values, old records, maybe. Yeah, old records. Okay, very good. So, now let's do it. We will simply say DF new equals DF join.filter. As you can see, it has autofilled. And very good. So, what we have done? We have simply filtered all the records which have null in their old dim customer key. That means dim customer key is not there. That means those are new records. So, that's it. That's what we want. That's what we want. Right? Very good. So, this is my DF new. And DF old will be opposite of it. I will simply say, and I will simply rename it. Or let's, DF old is fine. Okay. DF old is DF join is not null. That's it. That's it. That's it. That's it. This is also fine. So, I have separated DF new and DF old. Makes sense. DF old means that needs to be added. Makes sense. Like, this DF old is already added in our gold layer. So, that means we do not need to touch anything. Okay. We do not need to touch anything. And what we'll be doing here right now? So, we simply need to update only one column. What's that? So, basically, let's say we are not adding these records. That's fine. That's fine. But we are updating these records, right? What are we updating? So, let's say in the new records, in this one, for which we already have dimension keys, dim circuit keys, any value changed? Any value changed? Let's say customer name is changed. And there's already a dim key for it. So, we need to update those records, right? So, that's why we'll be updating these. Plus, one more column, which is called update date. We know that the create date column will not be changed. But update date column will be changed. And what will be the update date? Current date or current time. Makes sense. So, what we will do, we will simply go to here. And I will simply type preparing DF old. Okay. We need to prepare it. Yes. How? Because we need to make it beautiful according to our destination table. And the destination table, we do not have those old underscore those columns. We need to remove those. Plus, we need to create create date and update date columns as well. Makes sense. So, now, what we will do, we'll simply say DF old equals DF old.drop. Obviously, we need to drop all the old columns because we do not need that. But we will not drop this update it. No. Or we can drop it. Not a big deal. Not a big deal. Really. Yeah, not a big deal. Because update date column should be, you can say, updated. So, we can actually delete it. Okay. Okay. So, we can simply delete it. Let's delete all the things. Okay. So, I'll simply add a comment here, just for you. And we'll simply say dropping all the, dropping all the columns which are not required. Simple. And I have dropped this column. But I will also say, uh, adding update date column. Why? Because I want that column when we'll be just writing everything. Makes sense. Let me just make you understand. So, basically, we need to write our data in the same schema. We cannot change the schema. Okay. And for create date, we do not need to change anything. Okay. So, that's why I didn't run this command for now. So, that you can understand. So, even before doing this, let's remove these two things from here. Why? You will understand it. So, you know that in our end schema, end schema, like last stage of schema, has two columns. What are those two columns? Those two columns are update date and create date. Makes sense. If those records are there, listen to me. Look into my eyes. Okay. If those records are there, do we need to change the create date of those values? No, because those were created long way back. So, what we will do, we will simply rename the column. So, I will simply say renaming, renaming create date column. Makes sense. Old create date, old create date column to create date. Makes sense. Very good. I will simply say DF old. Very good. Now, this is done. Now, just tell me one thing. Will we update or will we change? Will we modify the update date? Will we change the update date column? The answer is yes. Why? Because this data is currently processed. Even if we do not have any new values, even if we do not have any modified values, but these records are processed right now. So, we will use the current value for our that column, which is like update data. Makes sense. Makes sense. Okay. So, now we can drop the update date column. Why? Because we will simply say, we will create a new column, update date. Update date. Okay. Because we'll be using something called as recreating update date column. Very good. I know these are not simple. But yeah, now you are understanding some of the things. Let's say you are understanding just 20% of the stuff. Next time when you will be just re-watching this part, you'll be understanding 60% of the part. Third time, you will understand 90% of the part. 10% should be there. That's the area of growth. Okay. That you will learn when you will be actually building it from scratch. Okay. Update date column with the current timestamp. See, it has written the code for me. And we have a function called current timestamp. It is similar to UTC now function in other fun softwares as well. In Azure and all, so it is called current timestamp. So, this is the value that we need to give. That's it. That's it. Okay. So, this is the value that is for current time. That means these values are processed right now. Okay. That's it. That's it. Now, we can have a look in DF old. And if I say DF old.is. Obviously, it is empty for now. But it will be full very, very, very, very soon. And the schema is perfect. Because we cannot play with schema. Schema is this: create date, update date. And actually, this create date is an integer. So, we need to convert it into date time. Because obviously, we cannot mess with our schema. Makes sense. We cannot. And from where it has just got this particular, you can say, number 1, 2, 3 and all? Uh, it actually got it from when we just did, uh, you can say, uh, uh, uh, from here, when we were just separating it, like from the above. So, what we can do, we can simply, uh, change it to the timestamp. So, how we can just do that? We can simply do it here. DF old equals DF old.with column. And then create date. And then we can simply say create date equals to timestamp. Two timestamp column of this. And then let's see what do we have here. Oh, it's still 1, 2. Oh, no, no, no, no. Okay, okay. And one brace was missing. These braces are man. So, now it is fine. Create date and update. Now it is fine. So, obviously, when we'll be just performing initial load, only then it will be a problem. Because when we have like, um, all these stuff, like, like when we have like table, okay, then obviously we can simply read that thing instead of reading it like this. This DF old, this one. And don't worry, I will just run this notebook multiple times so that you can actually understand everything. Don't worry, I'm here. So, now this particular thing will only be running one time. Because once the table is there, it will not pick this. It will automatically pick this. Makes sense. Very good. But obviously, you should have a schema fixed. So, just remember the schema. Now, this is the schema where it is. This is the schema. And we need to have exact schema in the destination. Plus, now it's time to prepare DF new according to this. DF new. Now, I will simply say preparing DF new. Okay. I will simply say preparing DF new. So, in order to prepare DF new, what do we need to do? What we need to do? What we need to do? In the DF new, what do we have? First of all, in DF new, we have this. We have this in DF.display new. This one, like all the columns. Okay. Makes sense. And all these old columns that we'll be just dropping. Okay. Makes sense. So, we'll be simply dropping these two columns. And obviously, we'll be dropped these two columns as well. Because these will be empty, these will be null. Bro, so we'll be recreating old create date and old update date. Now, just tell me one thing. What should be the create date for this particular data frame? And if you have answered right, that means you have understood more than 20% in first go. So, the answer is current timestamp. Because these are new records. That means these are created right now. And these are processed right now. Very good. So, let me just copy the code from the above for removing and doing all the things. I will simply say dropping these columns. Okay. And here, yeah, we need to remove all the columns. So, I'll simply say old create date as well. Makes sense. Yeah. Now, we will be recreating update date column, update date, and current date columns with the current timestamp. Okay. So, let me say DF old equals. Very good. Simply remove this. Perfect. And obviously, this is DF new. This is DF new. This is DF new. This is DF new. This is DF new. This is DF new. Very good. That's a real vault project, bro. Don't worry. Don't worry. Trust me, you will understand everything in just three runs. Like, in just three iterations. That's it. And in the fourth iteration, you'll be like, "This is just like piece of cake. Just a piece of cake." Okay. Because currently, you are understanding the concept plus you are implementing it as well. Okay. So, don't worry. I'm with you. I'm with you. I'm with you. Okay. So, let's run this. And let's see what do we have here. Perfect. Let me just show you what do we get in this. This DF new.display. And it should exactly match the schema. See, current timestamp, current timestamp. Very, very, very good. Now, one more thing in preparation of DF. What's that? In the dim customer key, what should be the value? It should pick as the starting value. We just discussed this. It should be the maximum of the previously loaded table. Now, it's time to actually add this dim surrogate key. And how we going to do it? So, obviously, we will just provide the dim surrogate key here. That is fine. But we need to simply add the last loaded value. Let's say in the last loaded value, we had 200 C, let's say 200 customers. So, we need to simply add 200 in all the values. Because it will start from 2001, 2002, 2003, 2004. Makes sense. Makes sense. Makes sense. Very good. Very good. And how we can just do that? Again, we going to perform a condition on the initial load flag. Because in this particular scenario, do we have any value? No. So, what should be the value? Zero. Okay. And in the other cases, what should be the value? The maximum of the value last loaded. Perfect. Very good. Now, you are understanding the concept. Now, I know. Now, I know. And just drop a lovely comment right now. I'm really hungry. I haven't eaten, eaten anything. Okay, just kidding. That's my love for you. Okay. So, now let's create the surrogate key. Surrogate key. And I will simply say, first of all, surrogate key from zero. Or let's say from one. Surrogate key from one. Okay. I will simply run this. And I will simply say DF new. Okay. And then I will simply say DF new. And here, if you see this dim customer key, this should not be there right now. Now, it will be added. Makes sense. Now, it will be added. So, it will simply say DF new.with column dim customer. Monotonically increasingly. Yay. So, done. Then we can simply see this data. DF new.display. Perfect. Now, we'll be adding the last maximum value. Okay. So, as you can see, dim customer key 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Very good. Now, we'll simply say text bold. Adding max surrogate key. Makes sense. Very good. So, how we can just add max surrogate key? We will simply say again, if init load equals to equals to 1. Then, obviously, max surrogate key will be, let's say, this is a variable. Max surrogate key equals to zero. Makes sense. If it is not, if it is not, then what we going to do? We going to load that particular thing. Really? Yes. And how we can just do that? We'll simply say DF max surrogate. Okay. And I will simply run spark.sql. SQL. And I will simply say max of dim surrogate key from gold. From data bricks kata.gold.dim customers. Makes sense. Just read the statement and just say, "Makes sense." Just tell me. "Makes sense." "Makes sense." "Makes sense." "Makes sense." What it will do? It will simply load the max of dim surrogate key from the table that is there. Obviously, it is not right now there. Because we are saying else. Because if it is not there, surrogate key is zero. It's fixed. But if it is there, it will simply grab the max value. Now, here's the thing. Because it will simply return the value in the form of DF. Now, we will convert the DF, which is data frame, into a variable. How we can just do that? We have a function called dot collect. Which will simply convert all the values into a list. We have take function as well. So, let's do that. Let me just write it here. Because we have else here. So, we have to write it in this particular box. Due to indentation. Let me just do that. Basically, it's fine. We can actually do it in the next cell as well. So, I'll simply say converting DF max surrogate to max surrogate key variable. Okay. Let me just do that. First of all, what is this surrogate key? It is called surrogate key. Surrogate key. Yeah, perfect. So, this is surrogate key. I was like, "What, what, what is written here?" Surrogate key. Yeah, perfect. So, now, obviously, we first need to run this. Okay. We can simply run this. Oops. What is the issue? Uh, table or view database. Okay. But it is one, right? Init load flag equals to one. Why it is going here? Table or view cannot be found. Obviously, it is not there. But why it is going there? Why? Hm. Maybe it is an indentation problem. If init flag equals to equals to 1. It is equals to equals to 1. Okay. Uh, I think I know the problem. What's that? Because, because I encountered with this problem before as well. So, the thing is, this is the widget. And by default, it is of type string. So, earlier, either we can simply quote it in the single quotes. But ideally, we should simply quote it in, um, integer. So, how we can just do that? You can simply say integer. Integer of this. And perfect. That's the power of debugging. If you have encountered with this problem in the past. See. Oh, it is gone, man. Now, just let's convert it. So, now we have this data frame. It is called max sur. Let me just first of all display it. We know that we just have one column, right? Max sur is not defined. Obviously, because it will not go here. Okay. It will not go here. But just remember, like, just imagine that you have just one column. It's called max surrogate key. Okay. And you need to just grab that particular value. So, in order to do that, we have something called as, I will simply say, max surrogate key equals df maxur. See, it is so smart. So, this is the code for that. And what it does? It simply performs a function called dot collect. And when we just run dot collect, it will simply create a list of all the values. Then we simply need to grab the first record, which is zero index. You already know the indexing. We have already performed in the silver layer. Then, within that, we have this particular key, which is called max surrogate key. Which is just a column. So, obviously, in our case, we have applied max. So, that means we just have one record. Even if you have multiple records, you can simply say 0, 1, 2. But when we using max, you will just have one record, right? Common sense. Common sense. So, now we can simply run this. Obviously, it will return error for this time. Because this is the thing that will be running only under this. So, now you have understood this. Now, I can simply cut and paste it here. Because it should be running under this area. Okay. And I can simply paste it. Paste this as well. So, that you will understand, like, what we are doing here. Otherwise, you will be saying, "Hey, what, what you are doing, man? What you are doing?" Perfect, perfect. Now, our DF new and DF old are all set to in, to be inserted into the gold layer. And how we can insert it? We can simply say, first of all, we simply need to apply a union between both DF new and DF old. Because both have same schema. Both have all the things that they should have. We simply need to apply union. That's it. We'll simply say union of DF old and DF new. Okay. Perfect. And I will simply say DFAL equals DF new.union_by_name. What is the difference between union and union by name? So, basically, union by name will apply the union by the column name. And it is a good practice. Obviously, we have same schema. But still, it is a good practice. Union by name DF old. That's it. Okay. And then, now, now, now, very good thing that I forgot. And by this code, it just suggested me that thing. We have max surrogate key, right? We have max surrogate key. Who will add that particular value? Me. So, add that value first of all. So, simply say DF new equals DF new.with column dim customer. And simply add this value, bro. Lit of max surrogate key plus column of this thing. So, now that 1, 2, 3 will be having that 200, 200, 200, or 2,000, 2,000 in the beginning, like 2,000 + 1, 2,000 + 2, 20, 2,000 + 3. So, now this time you have the updated data. Makes sense. So, simply say run this. And lit max key. Obviously. And what should be the output? Just tell me. If I just display it, what should be the output? Exactly same. Why? Because currently, the value of max surrogate key is zero. But for the subsequent runs, it will be max of the previously loaded value. Okay. Very good, bro. Very good. And now, let's perform, uh, union. Not union, jun. Okay. So, this is my DF final. I can simply apply union. And there is an error. What the error? Current date among customer ID, create date, update date. What is current date, bro? What, who created current date? Just tell me one thing. Who created current date? Who? It is called create date, bro. It is called create date. It is not current date. It is called create date. So, now everything is fine. Everything is fine. Let me just run all the things from the top. Which is called run all. And we should see the results here. Uh, now dim customer key cannot resolve. Okay. Now, what is the thing? Dim customer key. Mhm. Okay. And, okay, let me just check the schema of both the tables. What is missing? DF old.display. Let's see. Oh, we do not have dim. Why? Why? Why? Let me just check where did we miss that dim. Uh, customers key. Let me just check in the DF old. In the DF old. Where is DF old? DF old is here. Perfect. Okay. Perfect. Oh, I see. I see. I see. That what's the issue? We also dropped dim key. That we do not need to drop. We simply need to update the name. That's it. Because this dim key will be, will remain the same, right? Obviously. So, we'll simply say renaming this thing as well. So, renaming old dim customer key to dim customer key. Makes sense. Makes sense, bro. And perfect. Now, let's do run all. Now, let's see, bro. That's why slowly changing dimensions are the biggest headache for us. Now, what's the thing? Now, what's the thing? Dim customer key among. Now, also it doesn't have. Really? Are you serious? Are you serious? Are you serious? Let's see. [Music] Oh, man. Oh, bro. Where's Wait, wait, wait, wait. We We wrote it right. Dim. Okay. Dim customer key. Dim customer key. Okay. We have. Okay. Now, we have dim old customer [Music] key. Why it is not showing here? Why? DF old.drop. Oh, because we didn't remove it from here. I was like, "What?" Why? I was like, "Why are you serious?" Let me just do run all. So, yeah, I was saying that slowly changing dimensions are not easy to maintain. And that's why there's high demand of data right now. It is finally done. I want to see my DF final. How does it look like after after a great, great, great, great, great effort. Now, let's see. So, this is my table that it looks like. And obviously, we have 2,000 rows. And we have dim surrogate key as well. Dim customer key. And update date and create date should be exactly, exactly, exactly same. Because all our new records. I love you, man. I love you. So, this is my dim customer key. And it feels so good when we have finally created our dim. Finally. So, now it's time to apply the upsert condition. That is the final thing. And it is called update plus insert. And we already, like, all the hard work is done. Now, we just need to apply that upsert. Upsert command is automatically updated or let's say automatically inserted or updated for us. And Data bricks has made or you can say PySpark has made that code really, really, really simple to apply. Let me show you how you can just apply upsert. And just a quick question. What should be the column on which we need to apply upsert? What's that column? And the answer is surrogate key. Why? Because currently, we have surrogate key as 1, 2, 3, 4, 5, 6, 7, 8, and 9, 10. Okay. If we would be having some surrogates there, surrogate keys there, it will simply update those keys. Okay. Or it will simply update the values of other columns. If we do not have surrogate keys in the s, in the goal table, and we do have surrogate keys here, that is the case in our scenario, it will simply insert it. So, surrogate key is the deciding column as well. Surrogate key is just the replacement for your, you can say, primary key. That's it. Or surrogate key is a new primary key. That's it. Because it has simple number 1, 2, 3, 4, 5, 6. That's it. Now, let's actually apply our absurd condition. And I'm really, really, really happy that we have successfully implemented SEDD type one. And now you know what is the difference between a hobby, a normal project, between, and like, between a hobby project and a real world, real time project. Okay. So, it is very simple to just pick those Kaggle datasets and just drag and drop, blah, blah. This is data warehousing, bro. You have actually built star schema. Obviously, you're building right now. But you are building a star schema. Okay. That is the real world project. So, feel happy. Obviously, these things take time. So, now feel happy. And let's apply our absurd condition on our gold customers. Let's do it. So, good morning to everyone. It's 10:30 a.m. bro here. So, next day. And let's get started where we left off. So, basically, now we need to just apply the upsert condition, right? Upsert means update plus insert. Basically, upsert is a very popular term that you should be aware of. So, in the interviews, let's say interviewer will say, "Hey, how you can just work with upsert? And what is the slowly changing dimension associated with upsert condition?" So, upsert is nothing but just slowly changing dimension type one. And it is properly known as upsert because we just either update the records or insert the new ones, right? But, okay, how we can just apply that? It is very simple. So, first of all, let me just apply the text. Okay. I'll simply say SEDD type one. So, now, just tell you one thing. In order to apply slowly changing dimension, okay? In order to apply upsert, we should first have a table, right? We should first have some data on the destination, right? Right. So, again, we're going to use this flag. If flag equals to equals to 1. So, if it is this, then what we will do? We will simply write our data. Okay. And we will simply create a table on top of that data. And we'll simply call it as dim customers. Makes sense. This is just if it, if it is our like initial flag equals to equals to 1. So, now, I was thinking like, you can ask this question, "An, do we have any better way of doing it instead of flag?" Yes, we have. It's called Spark catalog. So, how we can just get that? Let me just show you. If I say, if spark.catalog.tableExists. And if I just say this thing. And if I just, let's say, oops. This auto suggestion. We should have a toggle button, bro, to turn it off. Seriously. Sometimes it is very useful, but sometimes it is very annoying. Very annoying. So, let's say I will simply print hello. So, what do you think? What it will return? Obviously, nothing. Because it doesn't exist. This table is not there. So, that is why we do not have any kind of output. Right? But if I say else, if I say, "Bro," it will return "Bro," right? Perfect. So, this is an alternative. My task was to just show you both the things. Because if we are using this thing, then we do not need to worry to maintain this particular flag. Okay. We do not need to change this flag again and again. But both the solutions do exist in the organizations. So, you need to be aware of this thing. So, let's say you are just joining an organization. You will be saying, "Hey, what is this flag?" And obviously, now you know. Because you are watching my videos. So, don't worry, bro. Basically, let's use this one for this case. So, that within one notebook, you will having both the things. So, that whenever you will be revising the stuff, or whenever you will be showcasing this project in front of any interviewer or anyone, okay? So, you would have both the things ready, right? Let's do this. So, if my table is not there, first of all, create that table. Else, now it's time to actually apply our upsert logic. In order to apply upsert logic, we need to first of all bring something called as from delta.table import delta table. What is this? This is basically the delta table object that we create on top of our data. Just to create an object, or let's say, just to create an instance of our table, so that we can simply apply the merge condition. Okay? Very simple. It's very simple. So, what we will do, we will simply say DLT object. Okay. Equals delta table.forPath. Oops. Delta table.table.t.table.forPath. We have two things: forPath or forName. We can also use forName. But I personally do like using forPath. Because it makes code easily reusable. H. So, now we need to simply need to pass the connection, which is spark session. Then location is this one. Ab fss. Okay. And this condition is also incomplete. Because there's no, um, location given here. So, we'll provide it. Don't worry. So, location is gold. At the rate datab bricks.et.dfs.co.windows.net. Okay. And dim customers. So, this is a location that will be created after this. Because obviously, in the first run, this condition will be running. And for the further subsequent runs, this condition will be running, right? Very good. So, now this is a location. Our object is ready. Now, we can simply apply upsert condition or merge condition. And how we need to apply? So, basically, this is our target, right? Where we need to just apply the merge condition. Because this is our main table. Yes. Perfect. So, I will simply say DLT object.alias. And I will call it as TRG. TRG stands for target. Then I will apply.merge. With what? With DF final. And I will also apply alias code source. Why? Because it promotes code readability. And it makes, let's say, there's another developer who is not much experienced. The person will understand. Okay, this is my target. This is my source. Okay. Then I need to say, what should be my condition? Which condition? Like, the joining condition. Because this is my merge. Then I will simply say target.dim customer key equals source.dim customer key. Very good. Just stop here. Because rest of the things I need to explain. Okay. So, this is my condition, right? This is really annoying, man. This auto suggestion. So, now we are good. Now, when we need to say, okay, we have applied merge condition on a column. But, an, how we will perform update and how we'll perform insert? So, bro, this is a good news for you. Everything is already automated for you. So, we have a function called.whenMatched.updateAll. What it will do? It will simply say, if dim customer key of my source is matching with the dim customer key of my target, then we know that we need to update that record. So, it will simply say update all. That means it will simply update all the columns of that particular record. Makes sense. Very good. Then we can simply say.whenNotMatched.insertAll. That means, if my key, dim key, is not matching with the target, that means that is a new record. And we need to insert it. And in the real world scenarios as well, sometimes we use multiple conditions. Let's say, when matched and something, something, something. So, in those scenarios, we prefer using SQL instead of using.

Python API, but in most of the cases, you will be using PISPA or Python API. So that's why I did this way. That's not a big deal.

And what are the other methods that we have? We have "when matched update". When we do not need to update all, so what we do? We simply say "when matched update" and we simply pass a subset. Subset equals to this. This is if you want to specify the columns that we want to update. That's it.

And once it is done, we need to write "execute", which is the most important thing to write at the end. Perfect. Makes sense. Let's provide the table location here as well. I will simply say "option path is this one". Okay. Makes sense. Perfect.

So, are you excited to run this? Let me just run that. What it will do? It will simply create a table at this location. Makes sense. Error. What's the error, bro? Is not a delta [Music] table. Okay. So, by the way, it is running directly this command. Why? Why? Why? Why? Why? Because you need to run this, bro. First of all. Oh, I see. Not really, because it is running this one. It's not a delta table. Hmm. Okay. Because obviously, if we are not defining any kind of format, it will basically take delta. That's for sure, because that is an optional thing. But it is not going here. If you just see carefully. Okay. But why? But why? Why? Why? Let's say, let me just comment it out and let's confirm it because I think so. Let me add "print testing". Let's see what command is running right now. See, it is directly going to the else statement. I think if condition is not working. But why? Let me add a brace here because sometimes it doesn't work if my brace is not working. Let's see what it will do right now. Uh, I think still it printed this, right? Uh, uh, uh, uh. Let me just change this value. Test. Let's see. Oh, yeah, it is directly going here. Let's see. I'm saying that, but in the above cell, we tested it, right? H, it was going to else. Oh, makes sense. Oh, man. So, basically, we need to switch the code. Just use your IQ. So, obviously, I'm just having some coffee. I think I need some more caffeine. So, the thing is, just, just, just use your brain. If table is there, if it exists, then obviously we do not need to create the table first. We need to switch the code. So, what I will do, I will simply copy this code. So, it is working fine. We simply need to switch it. So, what I will do, I'll simply cut paste it here and we simply need to paste it here. Perfect. That's it. That's it. That's it. That's it. Perfect. Perfect. Perfect. Perfect. So, simply use your brain. Use your brain because obviously, if our table is there, then obviously we need to upsert, bro. And we need to create the table only and only if we do not have the table. Now it's good.

Now let's run this. Now it should just create the table at this location. And now I will do one more thing. And after this, I would say your understanding of this whole solution that we have just built for gold customers will at least, at least two or three X. Why? Because now you will understand all the code that we have written. But why? Because now we have the data available in the destination. Earlier, we were just imagining. Now we do not need to imagine. Now we will actually see it. Okay, let's see. So, as we know that our initial flag load is zero now, because we have already, uh, loaded the data. Now, let's start running this code one by one and just observe what we are doing and just validating your imagination with the real code. Okay. So, we know that initial flag load is zero because we just changed the value. Now, what we need to do? We need to simply read the source. That is same. Makes sense. And this will be a quick test for our upsert condition as well, because all the records are exactly same. So, that means we do not need to insert any record. We simply need to update all the records. So, that is double testing that we are doing at the same time. Okay, let's do it. So, first of all, we'll removing the duplicates. That is fine. And we can simply see the data. It looks good. Yeah. Now, now is the thing. So, basically, basically, basically, basically, now this code will be running because initial flag load is zero. But why I'm seeing red here? Don't we have "dim customers key"? Do we have "dim customer key" or "customers key"? I think it is "dim customer key". Uh, "dim customers key" or "customer key"? Let me just check. Let me just check. Uh, "dim customer key". Oh, it is "dim customer key", not "dim customers key". Okay. Makes sense. Makes sense. Okay. Let me just remove as just a typo. "dim customer key". Makes sense. We can just remove it here as well. But it, it doesn't matter because obviously we are renaming it. But just for the readability, that's it. Let's run this and let's see. Earlier it was empty, right? Now you will see the data. Now the game will change, bro. See, now we have the data. And what's the create date and update date? What's the create date and update date? What we are seeing here? Obviously, the moment when we just created this. So, obviously, uh, now it is showing 27th because I just rerun the whole code because obviously I, I, I just restarted it today. Today, and I just left my code in the night yesterday. So, it is showing 27th. Makes sense. Okay. And don't worry, we have timestamp. So, like right now, it is 13:36. Okay. Very good. So, now we have this data. Earlier we didn't have. So, we were imagining what we will do. Now we will simply rename the columns. Just do it. Okay. And if you observe, we need to remove S from here as well. Just to rerun it. Perfect. So, now we will apply join with the old records. Very good. Let's do it. And let's see the data. How does it look like after applying the join? So, obviously, we should see old customer ID, create date, update date. Makes sense. And this is the "dim customer key" old. Now you do not need to imagine anything. Now you are seeing everything in front of your eyes. Now, what you will do? We have old "dim customer key" and all. Now we will simply filter down the records, right? New versus old records. So, obviously, in our case, no record is new. All the records are old. Very good.

Now, what we need to do? This is really interesting. Preparing DFold. So, what we are doing here? We are simply dropping these two columns. Okay. We are simply renaming "customer key" because we need to just sustain it. Then we are renaming "create date" with this one because we need to keep the create date as it is, because create date was 13:36, like date was obviously like today's date, but time was 13:36, right? So, we need to keep it as it is. But we need to simply update the this timestamp. Right? Let's do it. Let's do it. Now you will see the data here. And now you will see the magic. Create date will not change. Only update date will be changed. Like, not date, like exact timestamp. See, earlier it was 13:36. Now it is 13:40. But in the above code, in the previous, in the source, like in the target, from where we are pulling the data, it is 13:36, right? Very good. Preparing DF new. Obviously, we do not have anything. So, it will be empty. Very good. And this is my DF new. DF new is zero. And that's what we want. Okay.

Now, let's create the surrogate key. This is important step. This is important step. This is very important. Okay. Now, let's see what it has done for us. Now we need to adding max surrogate key. This is a great test because we should add only and only because obviously 2,000 plus something. And we do not have any records. So, we should not see anything. Let's see if our data is fine. Okay. Okay. Okay. So far so good. Let's see DF final. Okay. Let's see this. How does it look like? It has 2,000 rows. Very good. Now is the time. Okay. Let me just remove this. This is just a test for you. Okay. Let me just. Yeah. Let's perform our delta load. And now, what we need to do? We need to apply merge, right? And we should not see any more records more than 2,000. And all the records should be, um, processed like with the, uh, updated date, right? Let's do it. Let's do it. Okay. Okay. Okay. Almost there. Almost there. H. Very good. Now I will simply show you our results. Select a trick from database. scattera.gold.dim_customers. Very good. Let's see. Let's see. You should see new updated date. Earlier it was 13:36, 13:36. This time you should see 13:36 and 13:40. Okay. Very good. Let's see. So, rows are 2,000. That means it is working fine. And see, create date is 13:36 and update date is 13:42. Because obviously, it took 2 minutes as well, like while running it. So, now it is 13:42. Makes sense. Makes sense. Makes sense. So, our dim customers is running absolutely, absolutely fine. I'm really, really happy that we have just created this dim customers. And I have just shown you all the ways that you can just use. Like, you do not need to even manage this particular parameter. But yeah, you should know about this feature. Done. Very good.

Now, what we'll do? We'll simply go back to our workspace. And I will just do a small fix. What's that fix? It's not basically. But yeah, right way to just do the things. So, I will simply go to my silver tables. And I will simply perform overwrite instead of append. Because we do not want to add new data. We every time need to override the data. Okay. Let's do the same stuff with other as well. Why we are overwriting? Because we do not need to keep the data for silver layer. Because we are not maintaining. We are just keeping a layer called transient layer where we'll be just keeping the incremental. That's it. Makes sense. So, overwrite. Oops. Very good. Product. And now I'm really excited to just show you how you can just work with Delta Live Tables. And basically, before working with Delta Live Tables, just listen to me carefully. Because I have just talked about Delta Live Tables in the past as well. And people do not hear to me. Like, how they need to just configure the cluster. Then they just cry. Like, why our clusters are not working? And blah, blah. Just listen to me. I told you like how to just configure cluster. So, just watch that video carefully. Just watch those parts carefully. Do not skip it. So, and this is just for, like, those few people. I know my fam, you are really, really, really focused when you watch my videos. I know. I'm just talking about those new people. And you know what I mean. Okay. This is already override. Perfect.

So, now let's talk about Delta Live Tables in brief. Let's talk about. Let me just take you to documentation. It is very easy. Let me just give you a disclaimer. It is very, very easy. And it is built to make your life easy, bro. Okay. Just tell me one thing. Let's take our real example. You know that we spent so much of our time to actually create Slowly Changing Dimension Type One. Yes. And we were just like, so, so, so careful. We made some mistakes. But yeah, we were supposed to be so, so, so careful. Yes. One of the properties. There are so many properties of Delta Live Tables. One of the properties of Delta Live Tables is automating this Slowly Changing Dimension tables as well. Really? Yes. So, Delta Live Tables are like built to make your life easy. So, do not treat it like, hey, what is this? What is this? We do not know how to work with this, bro. Take your time. Delta Live Tables are new to us as well. But yeah, it is very, very, very easy. Let me just show you what do we have under Delta Live Tables. Okay. Basically, DLT stands for Delta Live Tables. It is basically an ETL framework. Okay. And what is the best thing about that? And what, what is new in that particular ETL framework which is not available in other ETL frameworks? So, this is a declarative ETL framework. What does it mean? So, basically, in a declarative framework, you do not need to worry about how we need to achieve that result. You just need to worry about what we need to achieve. That's it. How part will be taken care by Databricks or you can say Delta Live Tables. That is the definition. Simplest definition of DLT. H. Okay. So, let's say you want to create a storage dimension. We will simply tell. And don't worry, bro. I will just show you how you can just create Delta Live Tables for storage dimension. And we will be creating Slowly Changing Dimension Type Two. And bro, Slowly Changing Dimension Type Two requires more focus and more coding in order to create that. Okay. Because you, you need to maintain the history as well. So, everything will be automated for you. And you're going to love this. Trust me. Okay.

So, basically, if I talk about the backbone of DLT, let me just take you to the developer code. What do we have here? Um, materialized views. Blah, blah, blah. Let me just see. Hold on, bro. Hold on. DLT for database equal. Okay. DLT. DLT. DLT. H. Okay. Uh, maybe concepts. Yeah, yeah. These are the key concepts that I want to talk about. So, first of all, these are these streaming tables, streaming materialized views. Yeah. So, these are the basically the three pillars of Delta Live Tables. Obviously, these are two. Let me just talk about third as well. Don't worry. Streaming tables, materialized views, and there's one more that is called streaming views. Okay. So, we either create a streaming table, or we create materialized views, or we can create streaming views as well. So, we already know what are streaming tables. Streaming tables are nothing but just like unbounded tables which will be just expecting to receive data which can only be appended. So, these are the streaming tables. And then see, it is written here, append, apply changes. So, it can have one or more streaming flows, append, apply changes. Okay. Then we have something called as materialized view. What is the difference between materialized view and a simple view? So, basically, view just holds the query, just the query, just the logic. And every time we call the view, it runs the query in the real time and it gives us the result. But Matt view stores the result of that query as well. Oh, okay. So, whatever we are just querying, whatever, uh, query we are writing, it is actually running that query and it is storing that data as well in the physical table. So, that is the difference between materialized view and view. And streaming view is just a simple view, but it will be built on top of a streaming source which can be just appended incrementally. And it will give you the performance of exactly once as well. So, that is why it is a great feature to just give you the exactly once capability or idempotency. And you already know about that because in the view, it will every time run the whole query on the whole table. It will not run incrementally. But in the streaming views, it will run incrementally. And just for your reference, streaming views should only, it's not should, must only be built on top of streaming tables. So, that it can basically, it needs a streaming source, right? It needs a streaming source. It should be built on streaming tables, not on materialized views. No, no, no. Only on streaming tables. Or you can build streaming views directly on top of any delta table as well. It is fine. Because obviously, in the delta table, we maintain the delta logs and versions and all. So, it knows like, uh, which version we need to read. So, it is fine. Streaming views can be built on delta tables as well. But not materialized views. Not at all. So, this is just the high-level overview of that. And let's actually go to our notebook. And let's actually create something using DLT. And you can obviously, um, read this documentation if you want. I also haven't read this documentation much. But yeah, I just did some kind of investigation when I was just learning these things. And then I just started building this things. And it's fine.

Now, let's go to Python language reference and let's go to overview. So, this is the thing that we need to first import, which is called "import DLT". Okay. And these are the API references. So, these are basically the major APIs. Append flow, apply changes. So, basically, in order to create Slowly Changing Dimension, we use apply changes API. H. Okay. Then we have create, saying, create streaming table, expectations. Expectations are very important. We will simply use it. What are expectations? So, basically, whenever we create tables in production, okay, we need to apply some constraints, right? This is null, this is not null, this is unique. So, these kinds of constraints can be applied on the table using expectations. Makes sense. Very good. Then data set definition function. Let's go there. And this is the basic definition that it is creating. So, what it is doing? It is simply creating a Delta Live Table using a decorator. Now, what is a decorator? As the name suggests, decorator just takes your function and does some decoration and returns some result. That is the basic definition of decorator. And decorator is not aligned with Delta Live Table. Okay. Decorated something that you should know in Python as well. So, these are just the fundamentals. I think that particular time has come where fundamentals are fundamentals were always, always important. But nowadays, fundamental knowledge is the most important thing that you should know. Most important thing. Everything is built on top of fundamentals right now. Decorators and all these things. Everything. Okay. Then we have a function name. So, whatever name, don't worry, this is just a theory. I will just show it practically as well. Don't worry. So, this is a function name. So, we can pick any function name. Let's say you will say "def XYZ". So, whatever function name we will be picking, it will treat that function name as our table name. Okay. But if we just provide "table_name" here in the DLT.table function, it can take that table name and it will ignore the function name. So, that is up to you, whichever you want to pick. Then in the return query, you simply write whatever you want to return. That's it. This is your Delta Live Table. Trust me. That's it. Okay. Then we have something called as apply changes. Okay. You can definitely go. Mommy. So, you can definitely go through this because we will be just creating streaming table and all on these as well. Don't worry. And these are very simple when you just see. Whenever you read. And this is not like you are not alone in this. Whenever a person is seeing something for the first time, he or she will be like, what, what's this? It's fine. When I also saw this code for the first time, I was like, wa, bro, wait. So, it is fine. I know you are seeing at the red. DLT functions. Like, what is going on? I know. Have some water. Have some coffee. And just chill. I'm here, bro. All these things are very, very, very easy. You do not need to worry at all. And I will just show you what you need to do. Okay. What do you need to do? I will just tell you everything. Don't worry at all.

So, basically, now let's actually create our four streaming table that we can do. And you can simply see, create these three things: one is view, second is materialized view, third is table. So, what is the difference between, you can say, Matt view and table? So, whenever we use DLT.read, that is a materialized view. But when we use read_stream, so, basically, it is not using stream. So, it is also a Matt view. So, that is the difference between Matt view and streaming table in terms of definition and in terms of implementation, you can say. But if you just want to create a streaming table, then we just use read_stream. See, DLT.readstream. And is saying DLT module includes DLT.read and DLT.readstream function that were introduced and blah, blah, blah, blah, blah. Okay. Makes sense. So, let's actually start with it. And let's create our first streaming table or streaming view. And then we will be using that particular table as our source to create this thing. What is this thing? It is apply changes. Apply changes is your Slowly Changing Dimension code. And it is very, very, very easy. And enough theory is given. Let me just show you some practical things. Let me just take you there. So, let's go back to our notebook. And let's create our new notebook. And we will call it as, uh, gold products. And when you will be completing this notebook, you will actually say this thing like, Delta Live Tables are great. And it is automating a lot of automating a lot of stuff. So, now let me just tell you one thing. Obviously, I can just turn this cluster on, which is an all-purpose cluster. Okay. But in order to create our DLT pipelines, we cannot use all-purpose cluster. Why? Because they require job clusters to be running or let's say to be debug. Okay. But why we are just turning on our all-purpose cluster? All-purpose cluster will tell you if there will be any syntactical error. Oh, okay. So, yes. So, what we will do? We will simply create the pipeline first. Okay. Like the code for the pipeline. And then we cannot see it running. Then we will simply run it in the development mode. And then we can see that if anything is breaking, if anything is required or something like that. So, we have to first build the notebook. And then we will be running it. And let, then, then we will be simply debugging the notebook. Okay. Actually, you do not even need to run this. Because you can even use serverless as well. Because obviously, if you cannot see the output, what's the use of this? But why I turned it on? Because this was the issue from a lot of you. So, bro, listen to me carefully. What is written here? It is written four cores. Okay. Very good. Just for your kind information, it's not about free trial. It's about paid account as well. I have a paid account. So, I know about this. So, basically, by default, in each region, okay, in each region, in each location, we get something called as 10 cores by default. Okay. This is our quota of cores. So, it is using four cores. And in order to create a job cluster, we have to have to use at least how many cores? Eight cores. Because in the job cluster, we have to define a driver node and worker node. And each core, each, each node will take at least, at least four cores. But in all-purpose cluster, we can either define driver node. That's it. And driver just does all the things for us. Okay. So, now, just tell me, 8 + 4, how many? It will become equals to 12, right? And if you have 10 cores, will your pipeline be running? No. So, that's why I told you guys, whenever you want to run your job cluster, whenever you want to run your DLT pipeline, shut down this cluster. You have to turn it off. You have to confirm it. It is turned off. Plus, at the same time, you need to make sure that your DLT pipeline is consuming only and only eight cores. Because sometimes, not sometimes, by default, it uses 16 cores. And you cannot run your DLT pipeline, bro. You have to make sure these things. Okay. Very good. And I will not repeat this thing again. Okay. Good.

So, now what we can do? Best thing is, if you have a paid account, I do not know if it is available in free account. But yeah, you can simply try. Simply go to home and simply search quotas. And then you can simply say my quotas. And then you can simply search that particular VM. And I think VM name is standard_DS3_v2. Let's search it. How many do I have? So, I simply say, see, standard_DS3_v2. And let me just click on it. How many cores do I have? See, current usage is 40%. That means four out of 10. That means I also have only 10, you can say, cores available for this region, UK South. Makes sense. Makes sense. Very good. If I just click on it, you can actually say new quota request. If I click on this pencil, I can simply say new limit. I want my new limit is 20. I can submit it. And it will be approved very, very, very soon. In just few hours. So, you can try doing this. And do not cry if your pipelines are failing. But even if you cannot, um, upgrade this number, I will simply say, just turn this off. And make sure your DLT pipeline is using just eight cores. Just do some investigation, bro. Just do some investigation. That's how you learn. And I've already created dedicated community as well for help. So, you should ask at the right place, right? You should ask your questions at the right place. You are future data engineers. Or if you are already data engineer, you want to excel in data engineering, you have to learn how to debug the things, bro. Everything is new for everyone. You are not alone. Everyone is trying their best to just learn new things. Debugging these things, seeing errors, seeing errors for, let's say, at least, at least few weeks. That's how they are learning and growing. Do not be in that era where everything will be taught to you through dedicated things. No. Everything is changing. You need to troubleshoot. You need to debug. You need to explore. Makes sense. Take it positively, bro. Take it positively. I know we, we all see errors. I, I also see so many errors. And I, I, I, I cannot even say like, sometimes I just spend like days to solve it. But that's how you learn, bro. Use all the AI agents. Use all those things. Do your best. Do your best. Just say like, I will not eat anything if I, if I am not able to solve this. Just do that. And see the error will be solved in just few minutes. And I can just bet on this. You need to build that attitude, bro. Just build that attitude. Okay. Good. Awamba, focus here. It's not a philosophical class. Philosophical philosophy class. So, I was just cheering you up so that you can just try learning new, new, new things. That's it.

And now let's create our pipeline. Okay. The gold products. Okay. So, how we can just do that? Let me simply create a text box. I simply say "DLT pipeline". Okay. Makes sense. Now I will simply create first thing, streaming table. Okay. How we can create a streaming table? So, basically, we will create a streaming table on a source. What is our source? Silver.Products. So far so good. Very good. So, I will simply say "@ DLT.table". Okay. This is a decorator. Very good. Now, in this particular decorator, I can give the name. Let's say, "name equals". Um, I want to give name to my table is "dim_products". Makes sense. Then I can simply create my, uh, function. And I can simply say, um, "dim_products". Ideally, you should just keep the name same, just to remove any kind of confusion. But now I just mentioned that if you provide "name" here, then it doesn't matter whatever you write here. Okay. But if you remove this, let's say you are removing this, then it will take the table name as this one. Simple. Very good. Now, in this particular function, what we need to do? We need to simply create a data frame. Let me just take you to the documentation so that you can actually see what we are doing step by step. So, we are creating this thing. Uh, let me just go above. We are creating this, this thing. So, it is saying "streaming read on a table" because we want to stream read on a table. Basically, we want to create a streaming table on top of our data or on top of our table. Makes sense. Very good. I will simply say "df equals spark.readStream". Okay. Dot table. Because I have a table. And what's the table name? It's "databricks.ka.databricks.ka.silver.products". Makes sense. I think this is the table name. Let me just confirm. Customer silver. Oh, sorry. Product silver. Okay. Very good. Let me just reconfirm. Product silver. Yes. Product silver. So, this is my DF. Now, if I just want to perform any kind of transformation, I can, I can, I can simply say "df equals to withColumn" and blah, blah, blah. I can do that. Really? Yes. So, but, but now I'm not doing that because I do not need to perform any kind of transformation. I will simply say "return df". So, what it will do? It will simply return the data frame to me. And it will simply create a streaming table on top of this data frame. That's it. And see, this is declarative framework. So, you are not writing something called as "saveAsTable". You are not defining any kind of format. You are not defining any kind of mode. Everything is managed by Delta Live Table. And obviously, we need to import DLT. So, I will simply say "import DLT". And let's import some other libraries as well. From pyspark.sql.functions import *. Okay. Very good. Uh, yeah. Very good.

So, now, now our streaming table is ready. Now I want to create my streaming view on top of it. Let's say. So, you can do that. It's very simple. I will simply call it as "streaming_view". So, this time I'm creating streaming view on top of this table which is called "dim_customers". By the way, it is, it should not be called as "dim_customers" because this is not our dimh table. It is not our final table. We have just pulled or ingested the silver table. So, I will simply say "dim_products_stage". Makes sense. Okay. Very good. Because this is a kind of staging table for it. And I will simply say "@ DLT.view". Because we are creating a view. And I'll simply say "df_dim_products". And this is my view. So, just a view. Then what I will do? Now I can do anything. And obviously, we will simply create a data frame first of all. So, I'll simply say "spark.readStream.table". Okay. Now, you will say, "I'm lambda, this table is not created yet". Because we have not ran our pipeline. How we can just refer this table? So, we have a keyword called "live". Okay. And then we can simply say "live.dim_product_stage". That's it. That's how we refer to the tables in the Delta Live Tables. Because obviously, these tables are not created right now. Because these will be running in a separate pipeline. Okay. Makes sense. Then I will simply say "return on df". That's it.

Now, it's time to create the dimension table. And do we have any [Music] error? Proxy side. There was an exception while executing the Python proxy on the Python side. Okay. Read stream. Oh, uh, no, it's fine. Read stream. Yeah, it's fine. I think we do not have that particular, um, cluster ready. So, and we are just trying to run it. So, it is simply saying, hey, how we can just run this? So, obviously, just ignore it. So, now we'll simply say, and even if you see any errors, just ignore it. Because obviously, you cannot debug it till the time you are just turning it on. So, just ignore it. I'll simply say "dim_products". Okay. Let's create it. So, now, in order to create a dimension, we need to use an API called "apply_changes". Okay. And how we can just use it? First of all, we need to use something called as "DLT.create_streaming_table". So, what it will do? It will simply create a streaming table for us, an empty streaming table. And we will call it as "dim_products". Okay. This is our streaming table. Okay. Let's run this. Perfect. So, this is our streaming table. This one. Create streaming table. This one. This is our streaming table. Okay. Which is empty right now. Now we will use, uh, apply changes API. And how we can just use it? We'll simply say "DLT.apply_changes". And this time, we need to bring all those parameters. So, simply go to apply changes API. Okay. And this is the thing. Now, let me just explain you what are these things written here. It's very simple. Whenever you want to create a Delta Live Table, or let's say any dimension table, obviously, we need a target. What should be the target in our case? You have just three seconds. If your answer is that streaming table that we have just created, that is empty table, you are right. Very good. What is our source? You have another three seconds. Okay. So, if your answer is that view, because streaming table is the source for that particular table, your answer is right. Now, what is the keys? Basically, we need to apply a kind of upsert, right? Or let's say anything related to merge. We need to have a key column. So, what should be the key column in this particular case? Key column should be the primary key. And primary key is "product_id". Very good. What is a sequence by column? So, basically, in in Slowly Changing Dimension Type Two, we maintain the history based on the date column, based on the date column. But in our scenario, we do not have any kind of date column. But in real world, you will be having some date columns as well associated to your source. You will be having that. And it's not a big deal. We can simply go to silver table and we can actually add that particular, or we can just simply do it, bro. It will be good for your understanding as well. We will simply do it in just few seconds. Then just forget about these things because these are not necessary for you right now. This is important. "except_column_list". Let's say you have a date column in your source, right? Will you want that particular column in your target table in dimension? Obviously not. What you will do with that that date column? You do not need that. You will be using the date column created by the dimension, right? Not the date column of the source. So, we can simply remove it using this particular feature. It's called "except_column_list". Okay. Then this is the main thing. "stored_as_scd_type". If we write here "1", then it will simply say "1", like SCD Type One. If we say, uh, "2", then it will simply create SCD Type Two. We are creating SCD Type Two. So, that's why we'll simply write "SCD Type Two". That's it. Makes sense. Very good. Then you do not need to worry about this "track_history_column_list", "track_history_except_column_list". It is fine. So, now let's copy this. These parameters and let's fill these one by one. So, target is, target is this one. Or let's say you can simply write, uh, "dim_products". It's not a big deal. Okay. And data source is "live.dim_product_view". Very good. Now, what is key column? Key column is our "product_id". And I hope it is called "product_id". Let me just check. Yeah, "product_id". Very good. Sequence by column. It should be a date column here. But we do not have any kind of date column. So, we can simply pick, let's say, "product_id". It's fine because it will simply apply the sequence on that particular column. It's fine. Because you will be feeling confused. What we are doing? So, just to keep things simple in the beginning, that's why. So, we can simply remove these. And we can simply remove this as well because we do not have any date column. Then remove last two as well. This is important. "scd_type" as "2". That's it. Simply run this. And it will create the Slowly Changing Dimension Type Two for you. Now, again, same thing. Okay. Same error. It is saying, hey, how we can just do that? This, this. So, now it's time to actually create our pipeline. So, can you just imagine? We have created our Slowly Changing Dimension Type Two in just few blocks of code. Hardly we have written like 10 or 15 lines of code. And in other case, we just created a long notebook to just create Slowly Changing Dimension Type One. Not even Type Two. Just Type One. So, see how easy it is to work with Delta Live Tables. And how quickly you can just create these objects without need to worry at all. Okay. So, now what we need to do? We need to simply kill our cluster. Terminate. Okay. And just wait for it to be terminated. And it should be terminated like quickly. Okay. It is terminated. Now let's go to our pipelines. Let's create a pipeline. Create pipeline. And we have ETL pipeline. Sample ETL pipelines. Ingest pipeline. Ingest data from external sources in just few clicks. And you already aware about this. Simply click on this. And you know about this thing, right? We have already performed this. So, you can even perform these things from here as well. This time I'll simply say ETL pipelines. So, this is my pipeline name. I'll simply say "gold_products". Makes sense. Okay. Then this is serverless. It is fine. Product addition. Advanced. It is fine. Then pipeline mode. Triggered. It is fine. Then path. We need to simply pick the notebook. And it should be here in the gold products. Yeah. Here. Very good. And don't worry. We will simply perform a lot of things right now. Because I want to show you expectations as well. Because it is very, very, very important. Okay. Expectations. Yeah. So, this is the thing. Unity Catalog. We will simply use Unity Catalog. Default catalog is this one. Default schema is gold. Then compute. Now is the thing. Cluster mode is enhanced autoscaling. Simply say "fixed size". Number of workers. Just one. Just one. Okay. And then add configuration. It is very important. Then channel. Current. It's fine. Worker type is your VM that you are using. Simply pick this one with four cores. I know people would be using this one. Eight cores. You do not need to use it. Four cores. That's it. Then driver type. Simply pick this. And pick four cores. Extensively. Okay. Simply say create. So, your job cluster is ready. And you can see it is in the development mode. And if you just click on start, it will be started. But we are not going to start it right now. Because I need to add one more thing, which is called expectations. So, simply let me just take you to the documentation here. If you will see the expectations, expect here. It is. What are expectations? As I just told you, I am just looking for that cute diagram. Uh, expectations. Let me just search that diagram. Is very, very, very good. Expectations in Data Light Tables. Oh, yeah. This is a diagram. This is very good. This is very good. So, basically, you create your table. Makes sense. And in this case, our table is our dimension table. If we pass some expectations, what are those expectations? Expectations are basically the constraints that you want to apply on our table. Let's say I want to say "product_id" should not be null. And if it is null, throw the error or fail the pipeline or give some warning. So, basically, there are three types of thing that we can perform. First is warning. Second is drop. And fail flow. So, let's say this expectation is passed. Then obviously, it will simply go to continue processing. It is fine. But if it fails, then we have three actions. Either we can ask DLT to give me the warning. Okay. Or simply drop those records. Or simply fail the pipeline. See. So, we have basically three, three kinds of actions that we can perform on expectation failure. And then let me just scroll down. Then you will see, let's say I want to apply multiple, uh, expectations on top of my table. You can simply do that. And in the real world, we simply perform multiple expectations. So, this is a diagram for this. Let's say you are performing three expectations on table. Let's say you are saying "product_id" is not null, "price" is greater than zero, and one more thing. Three things. So, all should be passed. Then it will be going there. If any of the expectations fail, then again, three things: warn, drop, and fail flow. That's it. And how we can just perform this kind of expectation is very, very, very easy. We simply need to create a dictionary in which we'll be creating our rules. Like, rule one, rule two, rule three. Then we have a decorator called "DLT.expect". Which is by default, uh, providing you action as warning. Then we have "DLT.expect" or "warn". In which you can just simply write, okay, just send me the warnings. Third, we have "DLT.expect". Simply fail the flow. And you can simply see the thing here. This is my dictionary for all the rules. Then I simply say "DLT.expect_all". Yeah, we also need to use "expect_all". Because if we are using multiple expectations, we simply say "DLT.expect_all" or "drop". Because "all" word is for multiple rules. Let's perform. It is very, very important. And it makes your DLT pipeline much more, you can say, efficient in the production environment. Okay. Let's go to gold_products notebook. And let's perform [Music] some expectations. So, I'll simply say, uh, this is DLT.apply_changes. Okay. So, H. Okay. Makes sense. So, what I will do now? I will simply say, because this is my table which I need to just take care of. Or let's say I want to just perform all these things in the beginning of the stage. Because that makes sense. Because you do not need to worry about any of those things. Because you wouldn't want, like, the data should travel till the end and then it should fail. Block that data in the beginning. So, I will simply Okay. Expectations. So, I will simply add it here. And how you can just do that? First, you should just prepare the rules. And I will simply say "expectations". And "expectation" is, uh, my rules. This is a dictionary that you can create. Now, simply say "rule_one". Oops. "rule_one". And simply say "product_id is not null". Simple. Like SQL rules that you write in your SQL. Same constraints. "rule_two". "product_name is not null". Let's say. Then third is, do we have "product_name"? Let me just check first. Do we have "product_name"? Check in the silver, bro. Silver. [Music] Uh, yeah. And I will simply say, yeah, these two things. These two rules are fine. Okay. And let me just run this. And I can simply attach this cluster as well. Serverless. Or I can simply attach this cluster now as well. Because this is also created. So, I can simply create this Delta Live Table pipeline. Click on connect. So, now it is saying, you are now connected to a DLT pipeline. And blah, blah, blah. It is very good. And currently, we do not have any kind of, you can say, cluster. So, it is saying "current cancel". Current execution. Do you want to cancel cell? No. So, let's wait. So, now it is validating. It will take some time. It will take a lot of time to turn this cluster on. Because it is a kind of job cluster. And it takes at least, at least, I think, six to seven minutes to create this. But you should wait. Because it's always good to debug your notebook first before taking it to the pipeline zone. And then run it. And then see the errors. That's not a good deal. Simply turn it on from here. Once you create that cluster, then simply validate this notebook from here. And if everything is fine, then you can simply, simply, simply run the code. So, for now, what it is doing? It is simply validating everything. And at the same time, it is turning it on. Because it is not turned on. And if I just click on this "open in DLT UI", let me just click on and click on new tab. So, basically, click on this. So, it will simply take me to the, um, this area. So, now, currently, it is waiting for the resources. So, it is currently creating the resources for me. Okay. And then you can simply see the same thing here as well. So, I will wait for like five to six minutes. Once it is turned on, then I will simply run this thing. And till the time, I can continue with my code. So, these are my rules. Basically, these are my rules. So, I will simply say "@ DLT.expect_all". Or I want to drop the records if I, if I do have any.

Kind of, um, corrupted records. I will simply say drop. Then I will simply define what are the rules. My rule name dictionary is my rules. Simple. That is the thing that you want to do, and that's it. And you can even confirm this by going here. See, first we create the decorator for table, then we create the decorator for this DT expectations, and that's it. After that, you can simply write your DF. That's it. So, it's not actually complex. I would say it's just new. And whenever you see something new, whenever you just work with something that is new, you feel like not much confident, but it is not complex, it is just new. So, just absorb this knowledge and call it as a new thing instead of just saying, "Hey, it is a complex thing." Makes sense? Okay.

So, now let's wait for at least five to six minutes. And once it is done, I can simply run this whole pipeline, uh, in this area as well. And obviously, first in the debug mode. Let's wait. So, now, as you can see, that this step, "waiting for resources," is passed. And it took, I think, eight minutes. Let me just confirm. Yeah, eight minutes. It takes eight minutes. So, now, as you can see, I didn't see any kind of error here. "Waiting for resources." Yeah, I was expecting an error here because initializing, we need to just tweak whenever we just run the, um, this job cluster for the first time. But just a tip, just a quick tip. Let me just go to compute. Let me just show you something. So, if I just go to compute, you can see that I have even deleted the all-purpose compute because sometimes when we just terminate it, still you will see some error regarding to quota limit. And in order to eliminate that error completely, simply delete your cluster. See, when you just click on compute and all all-purpose compute, simply delete the compute. That's it. And wait for a few minutes, at least I would say ten minutes, because it takes some time to refresh that particular logs in the repository. So, delete the personal compute. That's it. And make sure all the computes, obviously job compute, you do not need to delete. SQL warehouse, if you have anything, just delete that. And that's it. And there should be only one thing, which is job compute. And this is the one. And this is the previous thing that I just saw because I was also seeing that particular thing because I didn't delete the all-purpose compute. So, I simply deleted it. And as you can see, now the active cores are eight. Otherwise, guys, these will be eight cores and all-purpose compute will be occupying four cores, and it will be creating that mess. So, simply delete this cluster and simply use this one. That's it. That's it. That's it. That's it. So, now you are good. Very good.

Now, let's see what is the error because now we are good with the cluster configuration. So, in short, delete all other clusters and just make sure only one cluster is running. And even before, um, you can say restarting your job cluster, make sure that you are like on hold for at least ten minutes. Just wait for ten minutes and then just give it a restart of that particular cluster because it takes time to kill the cluster and update the logs. Everything makes sense. Do not be in a rush. So, now let's see what's the error. Now let's see because now we are good. And DLD event log. Okay. So, one error is "fail to resolve flow due to upstream failure dim products view." Okay. Dim products, uh, failed to resolve flow due to upstream failure, dim products gold, dim products. Okay. Okay. And then "failed due to catalog." Okay. There are some failures. Let's see. And now we are good because we have this step, which is validate. And if you just want to validate the step instead of running, you can simply run validate and you can see the errors. So, one error is here, as you can see. Uh, it is saying "data brick silver product silver." What is the issue here? Uh, okay. This is our staging readstream.t. Okay. And these are the rules. Okay. And h, it looks good to me. Let's see what is the error here. Let me simply run this current execution. Uh, yes. So, it is simply stopping the flow, I guess. So, warning, info, all DT graph initializing. Okay. So, it was here and it threw some errors, and we simply stopped it. Not a big deal. You can simply see this here. "Failed to resolve flow due to upstream failure," which is dim products view. Obviously, that failed. "Failed to resolve flow due to upstream failure," which is this one. "Databrick kata dot gold dim products," which is this table because we are not using or referring this table. "Databrick scatter.go.prouctroucts products." I think this is the issue. Let's see where we have used this. Where we have used this. "Databricks dot." We haven't used anywhere. What error you are giving, bro? We are not using any kind of gold thing. Let me see. Validate. So, now I'm simply validating in my code again. And this time, it, you do not need to wait. See, because you already have some resources available. So, we will simply go directly to the error. And this is the event log. Okay. Okay. So, now it is giving me error for this thing. "Fail to resolve flow stage." Okay. It is saying "failed to analyze flow data bricks kata dot gold dot dim product stage." But what is this one, bro? Because we are not using this thing anywhere else. Anywhere. What code you are referring? Let me just check. We are calling silver. Okay. In the streaming, it is giving warning because "fail to read as it dim gold stage." Hey, wait, wait, wait, wait, wait. Dim product stage. Okay. It is saying, "Okay, fail to read dataset databrick scattera dot gold dot." Oh, I see. I see. I see. I got it. I got it. I got it. So, what is the thing here? So, basically, it is trying to create this particular table, which is dim product stage, in the gold because obviously whenever we just use live, so it uses this original schema, which is gold. And in our case, we picked the schema as gold. So, it is simply saying, "Hey, we cannot read this table." Okay. Makes sense. So, actual error is here, and something is wrong with this particular thing. So, here the thing is, if you closely observe, we didn't use "silver dot." We simply use "silver underscore." So, this is the root cause. Let me simply say validate for one more time. I was like, "What, bro? What, what gold table? I haven't written 'gold' keyword in the whole notebook." And let's see if we do have anything else. Uh, "fail to resolve databrick scattera." Okay. "Databrick scattera gold stage." "Failed to analyze blue gold stage." Okay. Is there anything else wrong? "Databrick scattera." Oh, spelling. Wow. Validate for one more time. Now it should be good. It was just a typo. Just minor, minor, minor thing. And I think it should be good. Let's see. Let's see. Because, see, I told you, in the DT notebooks, we cannot actually debug our code. So, obviously, we cannot see the root cause. So, you have to run the whole notebook and you have to see like where are the errors. That's the pain for now. But yeah, they'll be improving it. Don't worry. So, they should just allow running DT with the all-purpose job cluster as well. Like, they should just do this thing. Obviously, they'll be doing it because this is a very new feature. Everything is fine. And let me just click on DT graph and it should give me the graph. Very good. Very good. Looking, looking like a wow. And very good. Dim product stage, dim products view, and dim products. Very good.

So, now, in order to run this, I can simply click on start. And now it will simply run the whole flow for me. And earlier, we simply validated it if we have any errors or not. That's it. Now it is simply running our pipeline. And let's see if it is running fine. Let's hope, let's hope, let's hope. And I will simply show you the end table as well. Don't worry. And once it is done, then we need to create our fact table. And fact table creation is a task because you will learn how to apply joins, how to just bring the dimension keys and ignore the rest of the stuff. You will learn all those things. Don't worry. And we'll be creating our fact table notebook after this. Okay. So, this is our DT pipeline. It started running. And dim product stage, dim products view, and dim products. So, this is done. As you can see, it brought 500 records. Okay. Now, this is a view. Obviously, it will simply bring the new records. And all our new records. And here you should see 500 for one more time. Okay. And it is following all the constraints as well. That is the best part. Without any error. We didn't have any error for constraints. So, that was a good thing. So, now it is creating a dimension table for us. Everything automatic. Everything automatic. And it works exactly fine with the exactly once criteria. Let me just click on re-like start for one more time. And this time, you should see it should not process any record in the whole pipeline. Why? Because there's no data. There's no new data. And can you imagine like it is handling exactly once, it is handling upsert, it is handling slowly changing dimension, it is handling constraints, it is handling all your, um, keeping the history of the records. Everything within just few lines of code. And that's it. See, this time it brought zero records. It written, it, it wrote zero records. Perfect. That is the concept of exactly once. Very good. Very, very, very, very good.

So, now, what we need to do? We simply need to maybe productionize it. Or we can simply say, start. It is fine. Now we need to simply create a new notebook, which is our fact table. And what we will do in fact table? Because our dimension is there in gold layer, our second dimension is there in gold layer. Only fact table is missing. Let's bring it. Okay. Simply go to workspace and click on create and click on notebook. And simply say "gold orders." Perfect. Very good. And which cluster we'll be using? We'll be using serverless. Makes sense. Very good. Because we do not need to create another job cluster because we actually do not need here. Okay. So, I will simply say "data reading." Or I will simply first create a fact table. Fact orders. Okay. Now, let's say data reading. Fact table creation is the simplest step. Like, is the simplest step you will be creating in the dimension data model. It is very simple. So, how we can just do that? You will simply first of all read the source. And what is source? DF equals spark.dotsql. And then select a from silver dot. And basically, data bricks, data bricks dot silver dot order silver. Makes sense. Very good. Very, very, very good. Let's see what do we have here. Okay. So, just to give you a glimpse before that, let's give you an error. What's the error, bro? What's what's the error? SPQ SQL select from data bricks dots silver. Again, ancha. Again, we will add dot and database ET H. Sorry, database kata. Yeah, you are right. Database kata. So, now this is my orders table that we have in the silver layer. Now, just tell me one thing. And you should know about this. In the fact table, we only put the dimension keys and only the numeric columns. That's it. We remove all the other stuff. Now, this column is controversial. That should we just keep date column in the fact table? So, let me just tell you, in some solutions, yes, we can keep date column in fact table. But ideally, according to the rule of thumb, according to the definition, we should not keep the date column in the fact table. But there is no harm. There is no harm. Let me just tell you. Yes, there's no harm. So, how we can just deal with the date columns? We do not want to keep it in fact table. Simply create a dimension table on top of it. That's it. That's it. So, for now, we can just keep it as it is. No worries. But yeah, ideally, you should not. But in some solutions, yes, developers do keep date columns in the fact table. That's not a big deal or disaster. Don't worry. Okay. So, now we need to remove these columns or basically not remove, replace this column or this column. And this column. Basically, not order ID. This is my primary key. So, this column and this column. By the way, I can just remove this column as well. Not a big deal because obviously we will be having the combination of these two. So, for now, we will just keep it because this is just ID, not any contextual column, not any kind of, you can say, string column, nothing. So, now we will simply replace customer ID and product ID with dim customer key and product customer key. That's it. That's what you want to do. And how we can just do that? We will simply apply a join. So, let's create DF fact. Okay. DF fact. And I'll simply say "fact data print." Perfect. Perfect. Perfect. So, DF fact equals, uh, I will simply say spark.sql order silver. Yes. And basically, I have DF to. I am good. I do not need to rewrite it. DF.join. Okay. And now with what I want to join? I want to join it with, uh, I will first of all create the dimension tables. DF dim customer equals spark.sql select aix from databicks kata dot gold dotde customers. Perfect. And basically, I do not want a fix. I just want dim customer key. And obviously, the joining key column. And joining key column is customer ID. Okay. But I will just rename it as, uh, dim and then customer ID. So, that I can easily remove it. Okay. Let's do the same thing with dim products as well. [Music] Dem Pro product. Uh, perfect. Perfect. Let's run this. Let's run this. I can see one error. What is this? Dim product key. What is this? What is this? H, dim product key. Okay. So, what is the name of that? What is the name of key? What is the name of key? I think the name of the key is, is, is, is, what is the name of key? Let me just see. Let me just see because we didn't see the data, I guess, right? Let's see the data. And let's say select ax from databrick ka dot gold dot dim products. Let's see what delta live table has created for us. Let's see. And you will see that it has already maintained the history as well for us. You will see a lot of things. And it has done everything automatically. So, basically, as you can see that start at, start at column is keeping your this thing because we didn't have any kind of date column. But if you have any date column, then obviously you will see a kind of date here. This column tells you that, okay, this value was actually added at this particular stage. And then it will end at null because obviously this value is in use. So, this is the common definition. Let's say common thing about slowly changing dimension type two. So, you should have that fundamental knowledge. And that what is a slowly changing dimension type two? It is nothing but it keeps a track of your change, like when it was created, it's called start date. And when it was, let's say, updated or when it was not in use, so it is called endat. Makes sense. Makes sense. Makes sense. And we just pick product ID. So, that's why we are seeing this product ID. Okay. And this is our key column that we can bring it here. Obviously, we can just rename it as well, as you can say dim product key or anything like that. It's up to us. But it is fine. It's not a big, big deal. It is fine. So, I can simply say, um, product ID as dim product ID as, because this is just a replacement of your primary key. That's it. And then I would need product ID as well. Yes. Okay. And now it is good. And you know the hustle that we perform, like when we, when we just perform to bring or can say create this kind of table, right? So, everything is done automatically by TLT, Delta tables. Okay. Perfect. So, now let's first join dimcast. Okay. DF dimcus. Okay. On customer ID. But we have different column name. So, I will simply say DF of customer ID, dim customer ID. Perfect. So, TF dimro product ID, dim product ID. Yes. Perfect. Uh, yes. Okay. And we should just perform the inner join. Inner join will work. Obviously, but ideally, we should perform left join. So, let's perform left join. And how we can just perform left join? We know that how equals to left. And then here as well, how equals to left. And then what we will do? We will simply select the particular columns that we we want to select. Order ID, dim customer ID, dim product key, order date, quantity. Not really. I will select all the columns. And I'll simply drop just few columns. TF fact new equals DF fact.drop. And what do I want to drop? I just want to drop dim customer ID. And there's one more column, dim product ID. That's it. That's it. That's it. So, this is my DF. And I want to just show you how does it look like. Okay. And I'll simply say, oops. What's the error? String object has no attribute session. What do you mean by session, bro? What do you mean by session? Uh, wait, wait, wait. Um, oh, really? Yeah, I think we just need to remove this. That's why I do not trust this auto suggestion. Every time it just creates a mess sometimes. So, yeah, see. But yeah, sometime it is really quick. Sometime it is really, really annoying. Let's see what do we have in DF effect new. So, we are all good. Yes, we are all good. So, now what we need to do? We will simply write this data in our gold layer. But you know with which. Oh, okay. So, now we have customer ID as well. And dim product key. And if you want, you can actually drop the customer IDs as well, like this one. Because we actually do not need that. Why? This can be your interview question. Next interview question. Why? Because it doesn't make any sense to keep customer ID because now we have dim customer key, which is the joining key with the whole dimension. So, why do we need to just keep these IDs? Why? Obviously, no. There's no need. So, I will simply remove this as well. Uh, customer ID. Okay. And then product ID as well. Let's rerun this. And let's see what do we have. Perfect. Now our fact table is looking like a wow. Perfect. Only dim customer keys and numerical columns. And that's it.

Now, let's try to apply upsert condition or fact table as well. Why? Because obviously, bro, fact table needs to be upserted, right? If your fact table is small, you can simply do override. But ideally, it is not the case. We need to just, uh, apply the upsert condition on fact as well. So, I will simply say "upsort on fact." "Upsort on fact table." Perfect. I will simply say import DL, Delta.ts from Delta.tables import data table. Perfect. Same thing. Now, you know the steps. So, let's perform the if condition. If spark.catalog. um, table exist. Perfect. And table exist. And what's the table name? It's called databrick kata.g gold dot fact orders. If it is there, then we'll simply perform our upsert operation. And for now, let's write pass. Else, dfact new.write. Write.format delta. Perfect. Option path is this one. Perfect. Orders, fact orders. Okay. Very good. Very good. Very good. And dot save as table. And dot mode. It's fine. Because by default, it will take up and mode. It is fine. Now, let's write our logic for upsert. When we all know that's so easy. Delta table.for name. This time, let's use for name. Okay. Just to show you both the ways. So, this time, we simply need to give the name of the table. And then we will simply apply the merge operation. Dot alias. This is my target. Dot merge. Dact new. Alias source. And go away, man. Order ID equals to order ID. So, basically, now you will ask me this thing. What should be the, uh, primary key? So, if we do have natural primary key in our fact table, it is good. But ideally, it's not like it's not, you can say, rule of thumb that we have to have a primary key in fact table. Ama, what you are saying? We should have a fact table like primary key in the fact table, right? Uh, not as the native column. But ideally, fact table's primary key is the combination of all the dimension keys. Like this, this, this, this, like all the dimensions, all the dimensions, all the dimensions. If we have like deodor as well, then we will take simply combination of all the three. So, if you have natural primary key, it is good. But if it is not, it's not a problem. Because the primary key of the fact table is the combination of all the dimension, uh, keys. So, in this case, I will treat this as our dimension as well. The mod or mod or mod or mod or mod or mod or mod or mod orders. Then I will simply say and let's keep it capital. And trg dot dim customer equals source dot, uh, dim customer key. And perfect. When matched, update all. As we all know. And then obviously, dot when not matched, insert all. Then execute. Perfect. Let's run this. And let's see what do we get. We should see at least, I think, 10,000 records because in the orders table, we have 10,000 records. Let's query it. Asterisk from gold dot [Music] Fact orders. Right? Yeah. Okay. 10,000 rows. Perfect. Very, very, very well done. Now, let's rerun this. And let's see if it still returns 10,000 rows with up absort, with item potency, with everything. It's a double test. Again, let's see. Let's see. Fingers crossed. Fingers crossed. Let's see. Perfect. Perfect. Perfect. Perfect. Mission successful. Project is successfully built. Okay. Very, very, very well done. We have created our star schema. Okay. And now, if you just want to see, let me just refresh it. Now we have our all the things in the gold schema. Dim customers, uh, fact orders, dim products. All these three things are there. And what is this database internal? Basically, this is the catalog. Let me just show you what is this. If you just go to catalog, if you just go to databrick internal, just click on it. And this is the area where it manages all the things for your delta life table. So, if you get the question in the interview, you can simply say, we automatically get a catalog called databrick internal where it manages everything for our DT tables. Obviously, it is empty. Earlier, we had the access to this particular catalog as well. And we were able to see all the data and all the locations. But obviously, DT is managing it. Obviously, we should not interrupt their things because if we just do some things, it can break the whole data warehouse. So, that is why it's a good step by the data bricks. They have s, they have like simply made this available to system user. Earlier, like I was also able to see the data. So, now they have just removed everything. Very good.

So, now, if we just go to workflows, we can actually create our end-to-end pipeline. Really? Yeah. Let's create our pipeline. Let's create a new pipeline. Or we can even continue with this one as well. Bronze incremental. Okay. And we can even embed this particular pipeline in our new pipeline. Let me show. View. Create job. Okay. And if I just want to give any name, let's say "bronze bronze injection." Then I can simply say type is pipeline. There should be something called ELT or ETL pipeline. Let me just search. Pipeline. See. So, this time you can actually pick. If we do have any pipeline, we have "gold product pipeline." So, we can even pick this one as well. Or if you just want to put job run job, we can even put run job within this. And we have "bronze incremental." So, we can even embed the whole pipeline inside another pipeline. It's called parent pipeline. And rest of the pipelines are called children pipelines. Just for your understanding. But we will create the whole workflow in one. Okay. Instead of embedding it, I will simply go to workflows and I will simply first of all rename it. And I will simply say "parent pipeline" or you can say "end-to-end pipeline." Okay. Perfect. Let's see the tasks. So, we know that we have just these two tasks so far. So, this is our parameters file. This is our bronze file. Okay. That we just. After this, we need to perform the data transformation on the silver layer, right? Let me just do add task notebook. And this will be called as, let's say, "silver, um, orders." Okay. And let's provide the notebook path. And notebook path is "silver orders." Let's do that. And perfect. That's it. Create task. Very good. Now, let's do add task notebook. So, now you will say, "Hey, why it is running after silver orders? Why does that make any sense to run my silver orders first and then let's say run my silver customers?" Does that make any sense? Obviously, no. Because there's no dependency between these. This notebook can run independently without this notebook. So, we will run these notebooks in parallel instead of running the notebook in the sequence. How we can do that? Click on this. And obviously, I will simply call it as "silver customers." And dependency, instead of picking "silver orders," you simply need to pick the dependency as, uh, "bronze autoloader." See, now it is running in parallel. Perfect. What is the path is "silver customers"? Very good. Same thing I will do it with other pipelines as well. Save and continue. Very good. Add task. And I will simply say notebook "silver products." Okay. And then path "silver products." Very good. Perfect. So, now what we need to do? Dependent on "bronze autoloader." Very good. So, now you can see that we have created three tasks in parallel because that makes sense. Okay. Now, we do not need to orchestrate our regions by regions notebook because that is a static notebook. And there's no incremental data. So, it will be running independently. It is fine. Now, once our silver, uh, layer is done, then we need to run our, uh, gold customers and gold products in parallel. So, how we can just do that? How we can just do that? And we should only run the dim customers after silver customers. If that makes sense to you, because obviously this is our source, right? And products, dim products should run after silver. Makes sense. Let's do it. So, I will simply say add task. And ideally, let me just tell you one thing. We should wait for all the silver pipelines to complete first. Then we should enter in the gold year. This is my personal tip for you. And I will simply say add task. And this time it is a notebook. And notebook is "gold customers." Okay. And then path is this one. "Gold customers." Very good. And depends on all the silver, silver orders and silver products. See, now this pipeline will be running once all the three pipelines are completed. This way, we can just build pipelines like this. Makes any sense? Very good. Now, I will insert another task. And this time it is a pipeline. I will simply say pipeline. And it is a DT pipeline. We know. So, it is called "gold products." Very good. And this is a pipeline. Okay. Pipeline depends on silver customer, silver order, silver products. Makes sense. Now, it looks very messy. See, it. I know. But yeah, it is what it is. Because you need to just do all the, because you need to run both the dimensions in parallel. And there is no dependency. So, in order to optimize your jobs, you have to do it. But you can anytime click on it. And you will see the flow. Like what is doing here? Like what are we doing here? So, it is fine. Okay. Do not feel like, "Hey, there are so many threads. How we can just interpret it?" Simply hover over this. And you will see the blue colors highlighted. See. So, it is fine. Now, in order to run this particular, you can say, activity, we need to define a specific cluster. What's that? So, basically, we will simply go here. And pipeline is this. As we know, this is the pipeline "gold products." And depends on this, this, this. It is fine. So, now, if you just click on any, any, first of all, let's save and continue. If you click on any other activity, let's say "dim customers," it will ask you what is the compute. For now, we are saying serverless. Okay. Makes sense. Because serverless is already there. And this is the job cluster. But we can even ask it to run through job clusters. Because obviously, our DT pipeline will be running on the job cluster. So, we should pick job cluster for all the tasks. Because we are running these tasks, right? So, simply say job cluster, job cluster. And then job cluster. Or you can even pick serverless. Not a big deal. Because serverless compute can also work. But I'm just telling you for the better deployment and you can say best practices. Job cluster, job cluster, job cluster. Obviously, it will take some time to run this pipeline. But that's how we should run this. And this should work on job cluster. Perfect. And the last one is this one. Parameters. Very good. Now, we need to just add our fact. Okay. So, in order to add fact, we have to wait till both the pipelines are completed. Because, or let's say, both the activities are completed. Because we are referring, we are referring both the dimensions in our fact table. Common sense. Very good. Let's say notebook. And fact table. Fact orders. And path is this one. I know this is the best part of the project where we just orchestrate all the things. I know. I know. I know. And this will also be running. This depends on "gold products" and "gold customers." Very good. So, finally, congratulations. You have created this beautiful flow. This beautiful flow. See, all the things are done for you. First of all, it will read the parameters file. Incrementally load the data. Silver, silver, silver. Gold customers, gold products. This is like slowly changing dimension type one, type two. And this is our fact table. Very well done. This is end-to-end flow that we can simply click on run now. And obviously, it will take a lot of time. You can simply run it on your own machine and see the output. And if there are some errors, you can simply debug it. It's not a big deal. Because this is just a flow. This is just orchestration. An orchestration that we are performing. That's it. Nothing else. And it's time to sum up everything. But before summing up, let me just tell you one thing. Now, let's say your data warehouse is done. So, if you simply go to SQL warehouse. Okay. And obviously, we do not have any kind of SQL warehouse. You can simply create it. If you create, create a SQL warehouse, you will see cluster size, extra large. You can simply pick 2x small and simply create it. Then you will be able to see all your tables and warehouses. And you can actually query that particular table. Let me just create it for you. Uh, let's say 2x small. Uh, click on create. Uh, let's say cluster SQL cluster. So, it will simply create a cluster for you. And manage permission is honor. Yeah. Perfect. So, it will take some time to create it. And it is turned on. So, now what you can do? Click on this SQL warehouse. And then simply go to the maybe only my. Okay. This is good option. Only my SQL warehouses. So, this way, I can simply see only my SQL warehouses. But this is the cluster that you can see. It is turned on. And the size is 2x small. And this is the cluster that is designed specially to run your SQL workloads. And it will simply eliminate rest of the stuff. And it is built specifically to run your SQL queries on top of your warehouse. Makes sense. So, now let me just show you one more thing. If you just go to SQL editor, you will see everything here. See. And this time it is attached to SQL cluster, not the all-purpose cluster. And here is our catalog. If I just run anything, let's say I want to run select a bricks from, um, gold. Databrick kata gold dot, let's say fact orders. Run now. So, I will see the same stuff that I see in the SQL workbenches or you can say SQL server management studios. See. Perfect. So, this time I can actually feel that I'm building SQL scripts. And this, this is the latest update. You can say advancement that they have done in data bricks. This way, you can run SQL workloads efficiently. And let's say you want to share your gold schema with data analyst, data, you can say report builders, business analyst. They can simply use the schema. Okay. And they can simply run the queries here using databrick cluster or any kind of SQL server workbench. And let me show you another feature of it. If you just go here and just go to the queries. So, this is the area where you can just simply create the query. Same way that you have done recently. But this time you can actually save the queries. See. You can actually save this query. If you just go here and you can simply say, oops. If you just click on it, you can simply say "query one." That's it. So, it will be saved here. And you can just refer it later if you want. If you just go to queries. And you can simply say refresh. Let me just refresh. So, basically, if you just go to SQL editor and we should just need to make sure that it is saved. So, yeah, I knew it. So, as you can see that query one is unsaved. So, that is why it is not being shown there. You can simply say control S. And oops. You can simply say control S. And then it will simply say where you want to save this query. You can simply say save it here. Then your query will be saved there as well. Okay. Perfect.

Now, one more thing. We have an amazing feature. What's that? If you just click on this plus button, you will see that visualization. What's that? Click on this. It will simply create visualizations on top of your queries. And you can simply say save. And obviously, you need to pick some column. Let's say on X, um, bar. I just need to pick order date. And on the Y bar, I want to pick, let's say, quantity. And I want to create a line chart just to see the trend. Okay. So, it is just showing the quantity. Why it is straight line? Because obviously, the X-axis is timestamp. And it is recording records on seconds basis. Okay. And you can even select anything else. Let's say I want to pick year. And if I just pick quantity, then what I will do? I will simply pick quantity. And I will this time I'll simply create pie chart. And I will apply. Wait. X column is this. Group by on, uh, year. Okay. And X-axis will be my quantity. I'm not an expert in building charts. So, you can just play with it. Okay. And is always supported in the legacy. This. I think I just need to populate a lot of things. As like, you can just simply play with it, bro. I, I haven't built like much charts and all. So, you can simply build a lot of visualization from here. And I can simply pick any other, uh, visualization. Let's say bar chart. Yeah. So, this time you can see like in 2023, we had these quantities. The this quantity, these quantity, these years. So, you can simply play with it. I just wanted to show you that we have visualization feature available as well in data bricks. And you can actually create a dashboard as well. Dashboard. Ideally, we just create dashboard in PowerBI. Yes. But data bricks has launched a, just like, just launched a new feature in which you can create dashboard directly in data bricks as well. So, how you can just do that? Simply go to, um, partner connect. Where is partner connect? Maybe in the marketplace. Uh, here. Yeah. Partner connect. So, as you can see that we have PowerBI desktop. Okay. We have Tableau desktop. We have obviously DB cloud and fiveran. You already know. So, basically, let's say you want to create a dashboard in PowerBI. And how you can just do that? You will say, "Hey, we need to configure all the connections, all the data warehouse, everything." No, really. So, you can simply click on this. And you can click on connect. Okay. And what it will do? It will simply download the connection file for you. And this connection file, basically, this is a PBX file, which is your PowerBI file. It will have all the connections already filled for you. So, you do not need to actually do that hard work. That is why it is called partner connect integration. So, there are like so many things that are there in database. And obviously, a lot of improvements and updates that are coming every single month. So, let's try to say that our project is done. Because obviously, like there are so many features. Obviously, you can just play and explore and learn a lot of things. So, just to sum up, let's go back to our workflows. And let's see our beautiful pipeline, which is end-to-end pipeline. So, this is the pipeline that we have created. And I'm really, really happy that you learned a lot in this project. And yes, now it's time to actually flex in front of everyone that you have created this project. Because this project actually is a rare project, which is end-to-end project, handling incremental loading, slowly changing dimensions, star schema, your constraints, DT, all the latest things, everything. So, just create this project, complete this project. And I know you have completed this project. Now, drop a lovely comment on this project. And just share this video with others. And just share this project everywhere. And just tag me. I will try to just, um, drop a comment. Okay. So, this is the achievement for you that you have achieved recently. And it, this project was really, really like a milestone that you have achieved. Because this project was not easy. You have learned so many new things in this project. So, congratulations. And now it's time to say bye-bye. And I will see you in the next video. And you can click on the video coming on the screen. Because in this particular video, you're going to learn a lot of things in the world of data engineering. And and and you will become an outlier. So, just click on the video coming on the screen. I will see you there. Happy learning.