Transcription
If you're tired of wasting time on repetitive, boring tasks, you're going to love today's video. We're diving into Preact, a powerful Python based workflow orchestration tool that lets you automate the mundane and focus on what actually matters.
Imagine sipping your coffee while your scripts are running in the background on time, every time, and alerting you only if something goes wrong. Sounds like a dream. Let's build it.
Once you have your desired folder open, click on CtrlN to create a new folder. Alt D to access the taskbar. Type cmd and press enter. And this will open up the command prompt in the same folder without having to navigate to this folder.
The first thing we will do is we will create our virtual environment. We do that by using this command right here. Once that is created, you will see this folder that will show up in the same folder.
Next thing we'll do is we will activate the environment. We'll do that by using venv/cript/activate. And the way you can tell that it's activated is the parenthesis that says venv and then the path right here.
Now you can simply install the library. BIP install prefect. As I already have it installed in another virtual environment, I'm going to skip this step. All right.
So now it's time to check whether we have prefect installed correctly or not. We can do that by typing prefect version inside of our virtual environment window in command prompt. And you can see this gives you the version of the Python prefect version that is installed right now.
Now that we have confirmed that prefect has been successfully installed inside of our virtual environment, let's switch gears and look at a real life example. All right.
So let's say every morning you open Excel. So let's open Excel. We have the sample sales data here that's updated on daily basis. You filter the data. So from sales invoice, we are going to filter completed once. Then you export the data. So what I can do is I can just copy and paste this data inside of a new file and then save it as a new file that is filtered daily sales update that is always created at the same location. But keep in mind that it only contains rows where we have sales as completed.
Once you have this file generated, you then send this out to your manager on a daily basis. Boring, right? But pretty fact, here's how we automate that.
So first let's create a script on a Python. So let me open up VS code and I'm going to start adding the code step by step and then I will explain each step. So first add our imports. So we are going to import flow and task from prefect. We are going to import pandas sbd for data analysis. We don't need this one. So I'm going to remove this one. We need win32.com client as win32. This is what we are going to use to send out email through our outlook client version and then we import python com.
So the way prefect works is it is divided into tasks and flow. So tasks are basically smaller portions that are ultimately called within our flow. So let's start adding some data so you can see what I mean. So let me add a small task here. So this is going to read the data from the Excel file and return the data frame. Now what we can do is we can add a decorator. So we can say add the task because this is a smaller task portion.
Similarly we can create another task. Keep in mind that the decorator that is above that says task. So this task will basically filter our data frame that is loaded and it will apply filter on the sales invoice column and only retain rows where we have completed as our value or status. Then we export it to Excel in this specific location. So we have the file name as filter daily sales update.xlsx and we have index as false.
Let's add another step that will actually send out the email. So let me add the code here. So similarly we use the task decorator. Then we use send email. But this is very standard generic format for sending emails. Then you add the two subject body and the attachment. So for attachment I'm using the same file that is created from here which is the filter data for the daily sales data. And this will only display the email instead of sending it. If you want to send out the email as well, you can simply do mail dot send. For now, I'm going to keep this commented out because we want to see the email that is being actually generated. So we can see if our code is actually working.
All right. So time to add the actual flow. So we add the decorator here. So we say add the flow. This is what actually going to club all of these three tasks and run it together. We are first calling this function read data and we are saving the loaded data frame inside of data variable. Then we are calling another task filter data and then we are saving filter data inside of this and finally we call the final task which is send email and we feed it the filter data frame.
All right. So let's save and close this file. The file name is test_flow. py. So file is located here test flow as you can see here. Now let's schedule this so it can run at specific time every day. So you can enjoy your coffee instead of babysitting scripts all the time.
So let me open up a command prompt here. So I'm going to say cmd and this opens up the command prompt at this specific location. So let me activate the virtual environment. This is very important as you are doing all of this inside virtual environment. Must be activated to perform anything. Right? This is 12:34 a.m. right now and I want it to run at 12:36 a.m. So, all right, let me paste the actual code.
This code provides prefact instructions on how to set up deployment and creates a YAML file in the same directory. YAML is a kind of JSON file with configuration instructions. All right, let's break down this command into smaller portions and try to understand it. So the first portion we have that says prefect deployment build. This is just we are calling prefect library and then we are saying deploy the build.
Next portion from here till here we have the actual name of the code. So we have test_flow. py and then within that we are calling daily task which was the actual flow. So if we see here this is the flow that actually calls the three different tasks and merges into a one flow. Then you see daily script run. This basically just tells it that this is going to be a daily script running.
Then you see chron. Chron is basically a Unix based job scheduler that helps you automate and schedule tasks. It has a specific pattern. So here 36 means minutes, zero means 12:00 a.m. And you also have other different options you can work with. And finally, you have a time zone. This is important because prefect will pick up the default UTC. So it's very important that you configure the time zone you're in. Right.
Next we use another line of code that actually defines the YAML file that has been created. Right. So next important step is that we create a agent pool. So we start an agent pool which is what is going to monitor all of the workflows created here. So you can see it says prefect agent agent started looking for work from work pool. So it should pick up a code in few minutes.
So, let's keep Outlook open. This is the Outlook I have. I'm going to make it a bit smaller so we can see the screen right here. So, it's going to pick up. So, you can see it has picked up the flow and it says a bunch of things. I'm going to make sure this file is closed because this file is being used inside of this flow. By the end of this we should have a email created with the data we added in the code ready to be sent out. So let's wait and watch what happens.
Right? As you can see you have some progress here. Uh it finally says finishing state completed and you can see tasks that are being run. So you have task one, you have task two, task three and finally the flow that is here says all states completed. Now you can see that the email has been generated with the information we gave it. So we gave it recipient, we gave it a test subject, we gave it a test email body and it has correctly attached the attachment as well. So let's open up and see. This should be the same attachment we give it. Right? This is the same attachment that we gave it.
Just make sure that this window that is the prefect agent continues to run and if it's scheduled on a daily basis at a specific time it will continue to pick up the flow from there as long as this specific window is open. All right.
You can also access the cloud login. So within the virtual environment you can simply say prefact cloud login and it will give you a bunch of steps you can follow from the instructions from here. And once you're set up you will see a dashboard similar to this one. I've already done it. So I will leave that step for you to set up. If you are having an issue setting up the dashboard, let me know in the comment section.
As you have seen, Prefect gives you dashboards, logs, and notifications capabilities as well. You will instantly know if something fails, and you can even set up alerts via Slack, Teams or email. You just save yourself hours of boring manual work each week. So imagine automating daily Excel reports, sending emails and other stuff.
If you found this video helpful, smash the like button, subscribe for more automation hacks, and comment below what's the first thing you're going to automate with Prefect. Till next time, happy coding.