Transcription
Web scraping can be essential for obtaining the data you need for your applications. If you've ever found yourself battling the wild beast of CAPTCHAs or navigating the treacherous terrain of IP bans, you're in the right place. This is a guide to web scraping in 2024.
As technology evolves, so do the tools and strategies we need for effective data collection. Today, we're diving into the essentials of modern web scraping. For many applications, it's necessary to get clean, comprehensive data, but many websites make it challenging to scrape their data.
To help navigate the complexities of web scraping, I consulted with the team at Bright Data, a leader in data solutions. In this video, I'll reveal the top five things you need for effective web scraping in 2024. Then, I'll take you through a detailed tutorial on using Bright Data to scrape information, showcasing the practical application of these tools. Bright is sponsoring this video, and I truly believe they have some amazing tools for web scraping.
But before exploring advanced tools and techniques, it's important to grasp the fundamentals of web scraping. Puppeteer is a powerful tool for developers looking to collect data from the web. Let me show you how to use it real quick, and then we'll talk about the additional things you'll need to know for modern web scraping.
In this example, we'll start by importing Puppeteer and defining an asynchronous function to scrape the title from a given URL. We'll launch a new browser instance, open a new page, and navigate to the URL. Then, we use `page.evaluate` to run JavaScript in the context of the page to retrieve its title. Finally, we log the title to the console and close the browser. This basic example demonstrates the power of Puppeteer for web scraping tasks.
However, the ever-evolving landscape of the internet presents new challenges that require advanced solutions. So here are the top five things you need to think about in 2024.
Imagine trying to sneak into a club, but the bouncer knows your face, your friends, and even your cat. That's your average website trying to block scrapers. Many websites will kick you out once they recognize that you are trying to scrape their data. Enter residential and mobile proxies, the digital equivalent of a master disguise kit. These proxies let you blend into the crowd of legitimate users.
You can't just scrape a bunch of data from a single computer or even a bunch of servers. If you try, you'll just be blocked. If you want to get a lot of data, you want to scrape from residential IPs, as those are harder to block.
Next up, the bane of every scraper's existence: CAPTCHAs. Those puzzles that ask you to identify traffic lights or storefronts test if you're a human or just a very confused robot. Companies are always coming out with better CAPTCHAs, so you always need to be one step ahead. Data unlocking tools, such as Web Unlocker, automate the process, ensuring continuous data access without manual intervention.
You've always needed web scraping scripts, but the most effective way to write them has changed over the years. Writing scripts for web scraping is like an art form and can sometimes be tricky to determine the best way to get the data you need. Tools like Playwright, Puppeteer, and Selenium are helpful, and I've also found Bright Data's scraping browser and their web scraping IDE can make things even easier.
I'm going to show you how to use the web scraping IDE later. Once you've got your data, you need somewhere to put it. Integration capabilities are your data's best friend, ensuring it's not just collected but also usable. This requires scalable solutions that can handle large volumes of data.
Last but not least, scalability. Your web scraping operations must be able to grow with your data needs, supporting large-scale data collection without compromising speed or accuracy. One thing I like about Bright Data is that they can scale things pretty much automatically.
You can certainly implement all these things yourself, but Bright Data makes it simple to deal with these five essentials of web scraping. Bright Data has a robust infrastructure featuring a vast proxy network of over 72 million residential and mobile IPs. This extensive network ensures reliable and undetectable access to data, which is helpful for bypassing site restrictions and maintaining anonymity online.
They also have advanced unblocking tools. Here's how Rafael from Bright Data describes the unblocking tool: "If you're using our unlocker and you send a request to a page, an HTML page that you would like back, our web unlocker can even open a browser if it needs to solve the CAPTCHA, get the HTML back, and send it back to you. So you can even get, with a simple curl command, information like Amazon products or any other website that is actually... if you send a curl, you usually get a bot block or no JavaScript or something like that."
Now, let's see these tools in action with the tutorial.
Let's imagine an online pet store wants to ensure their prices are competitive with the prices on PetSmart.com. I have this demo pet supply store I call Pet Brigh, and you can see it has all these products, and they all have prices, names, and companies.
Let me refresh my page so we can get the current prices. Let's just look down here at some of our prices. Look at this: this is $84.49. I'm going to copy this item so we can see how much this costs on PetSmart.
In our store on Pet Brigh, it's $84.49. Let's see how much it is at PetSmart. So let me just paste in the description here, and you can see here it's $81.99. So the same item is $81.99 at PetSmart, but in our store, it's $84.49. It's more expensive. Nobody's going to buy from us; everyone's going to go to the cheaper PetSmart.
So I'm going to show you how you can use web scraping to quickly scrape all the information from the products at PetSmart. Then we'll also see how we can use that scraped data to automatically update the prices and the information in our sample pet supply store. We want to ensure we sell for the same price or lower than PetSmart so we can get the business.
Let's go over to BrightData.com, and I'll go over to my user dashboard. If you don't already have an account, you'll have to create one. What I'm going to go over to is web data because we're going to be trying to get some data from the web, and I'm going to use the web scraper IDE.
This allows us to create a web scraper right within the Bright Data website, and they have their own IDE. So I'm just going to get started and then automatically create a new web scraper.
Now, it could start with one of these templates. All these templates will show how you can already get data from these different websites, but I'm going to start from scratch because we're going to scrape the PetSmart website.
This web scraper, or what they call a collector, is broken up into a few different sections. We have the interaction code; this is the code that's going to interact with the website. Once we've gotten the website to the correct spot, we can parse information from the website.
We have the parser code, and you can see in this example here it's showing that first we navigate to the website, then we parse the data. We call this parse function; it just runs the code down here. In this case, it's going to get the H1 tag and all the links and put it into the data variable. Then we call the collect function, and that's to collect our final data. We'll be collecting data that we can then output as JSON or some other data format.
This is just an example; we're not going to use this one, so I'm going to delete all this. I'm going to start by specifying the country. If you don't specify the country, then every time the code runs, it could be coming from a different country. Some websites will display things differently depending on the country of origin of the request, so we want to make sure the request is always coming from the US.
Then we are going to create a URL. I'll say `let URL` because we're not just going to go to just petsmart.com; we want to go to a specific part of petsmart.com. Let me go over to PetSmart.com so you can see what I mean.
We have different categories here, and for our web scraper, we are going to allow the user to select what type of animal they want and then what category. You can see in the dog category, we have food, treats, supplies, and for cats, we have food, treats, and litter supplies.
For this example, we're going to do dog and food. If I go to dog and food, we can see in the URL here a lot of this is just extra information I don't need. We can see it's petsmart.com/dog-food.
Now, everything I'm teaching you will work for pretty much any website. I'm going to show you how to get the information we want from the PetSmart website, but you can use the same principles to scrape data off pretty much any website you want.
So the first step is to figure out the URL that you want to go to and if there are any patterns with the URLs that can help you. I'm going to allow users to input the animal and the category when they run the scraper. Let me show you how we're going to do that.
So let me go back over here so we can see now the URL. It's petsmart.com plus now this is a variable input: `input.animal` and so the user is going to input an animal, and then it's going to be `/input.category/`. So that would be `petsmart.com/dog/food`.
How do we get `input.animal` and `input.category`? Well, we define it down here. We want to add an input parameter, so I'll put `animal`, and that will be a type of string. Then I will add `category`, and this will also be a string.
I can add some sample data right here. If I put another input, the sample data would be dog and food, but when you run the scraper, you can put any animal in any category as long as it corresponds to the right categories on petsmart.com.
Now that we have the URL, we are going to navigate to the URL, and we're going to wait until it loads. Then we're going to do it in a stealthy way. The truth is, I don't know completely what this means, but a lot of the examples have that.
So after we navigate to the URL, we're actually going to go to the next stage. See how it says collector stages? We're on stage one, which we're going to call URL Setter. We're going to make it stage two. It will become obvious in a little bit why we need different stages, but this is going to be called get links and pages.
We're going to potentially rerun the stage a few times, and we don't need to rerun setting the original URL. That's why we go to a new stage that can be rerun a few times.
We're going to call the next stage function and pass in the input parameters. We're going to pass in the URL and the number of pages. We have `input.animal`, `input.category`, and `input.numberPages`. That's something that when you run this collector, you can put in how many pages you want to scrape.
If you don't put any number here, it will scrape all the pages, but you may not want to scrape that many. So we also have to add another input parameter right here called `numberPages`, and this is going to be a number. This is not going to be a required field, so let me edit that, and it's not going to be required.
We're passing that on to the next stage because we're going to use it to determine how many pages we're going to scrape. I don't need this default data. We're going to set the country to the US again, we're going to navigate to the URL, and then after we've navigated to the URL, we are going to scroll to the bottom of the page.
This is often helpful because you want to make sure all the information is loaded before you start scraping. Some pages don't load things until you scroll to them, and we're actually going to try this out, and we'll be able to see it all happening in this preview right here.
Before we parse, we want to make sure everything is on the page. Let me show you what this page is going to look like. The dog food page—remember, we want to get all the product information from the dog food category.
Now these are all categories, so we don't want this. If we scroll down, okay, now we're starting to see the products. You can see there's a bunch of products on this page, and there are actually 50 different pages we can get. We can hit next a few times, so this is why we want to define the number of pages we want to scrape.
If you want to scrape one page, you'll get all the products on this page, and you can imagine it becomes a lot of products if you're going to scrape all 50 pages. So first of all, we want to make sure this loads, and we want to wait until the page loads or until a specific element on the page is loaded.
When you're doing web scraping, you often want to go into the developer tools. I just did Command + Option + J, but you can also do View > Developer > Developer Tools or JavaScript Console. If I click here, we can select different elements, and if I go here, we're going to find something at the bottom.
Well, see, we see this page footer at the bottom. We can assume that if the page footer is loaded, then all these products have finished loading because the page footer is way down there. So I'm going to wait until the page footer loads.
These are all just CSS selectors. Once the page footer loads, we are going to parse. We want to parse all of these links. We're not going to get the information quite yet because each of these items, if we click on it, you can see it loads the product page.
Here, there's going to be more information, like there's going to be a size, there's going to be the price, you got the brand, and the title. Now, a lot of this information is on the previous page, but you see you don't really see the real price until you click into it.
See, this is $3.09 over here; it has a range, and it may not be exactly what it is when we go into there. So basically, what we want to parse right now is the link to every product page.
We also want to figure out the exact number of pages there are. In this case, we can see that there are 50 pages. So let me show you how we're going to do that. We're going to run the parse command. Anytime the parse command runs, it's going to run this parser code down here.
So here's some parser code that I already figured out. Let me show you how I figured it out. For the links, we're looking for the CSS selector named link. If we go over here, we're just going to go to the first one up here, and we can see if we scroll up, we can kind of see where the link is, where the anchor element is.
We can see it's class name link, so every single one of these elements has the class name link. We put that into an array, and then here we are going to get the URL and eventually put it into a string.
To figure out how many pages there are, if we click right on here, we can see this has the class page last. That's why we look for the class page last. This actually shows up two times on the page, once at the top and once at the bottom. We just want one of them, in this case, the second one, and then we want the text, which is going to be 50.
So there are 50 pages. Once we have the pages, what are we going to do? Once we have the links and pages, well, let's put these into a variable. So we'll do `let linksPages = parse`.
Now, let's worry about the pages in a second. What are we going to do with the links? Well, we want to, for each link, actually go to the next stage because in the next stage, that's when we're going to collect the info.
So I'm going to say `links.forEach`, and then we can see we're going to run for each link. We're going to call the next stage function, and we're going to pass in the URL.
Okay, but remember, there could be multiple pages. So what are we going to do about those pages? Well, first, we're going to see if the number of pages was passed in. If the user set how many pages they wanted to collect information from, if that was passed in and the number of pages is less than the pages on the page, then we'll just set the pages to the number of pages.
We may not want to parse every page; we just want to go to that many pages. Now, if there is more than one page, we're only going to run this code if there's more than one page. If there's just one page, we just go right to here, and then we're done.
But if there's more than one page, and I'll explain this in a second, we're going to loop through all the pages, and for each page, we are going to rerun this whole stage. We have to figure out the URL to go to.
To figure out the URL to go to, we can just click on this and see how the URL changes. If you look on this, this is page two; this is page three. The only thing that's changing is this start. See this start? There's a key-value pair, and the start is which element this is.
Now this page is starting at the 36th element, and there are 36 items on the page. This next page is starting at element number 72, and there are still 36 elements on the page. So that's why we take the URL, then we add how many elements are on the page, and then we get the start element.
The start element is just going to be `i * 36`, so whatever we loop on time 36. There are actually a lot of other things on this URL, but you can just kind of play around to see if you even need this.
If I take all this off and I just go to the page again, it loads the exact same page. All the other information was not important, so I don't need to have it on here.
Then we're going to say `isRerun = true`. So every time, the only the first time this code is run is going to rerun the stage for every page because we don't want to rerun the stage for every page. Only the first time we're going to rerun it all, and that's why this makes it so only does this part one time.
So it's going to run all the pages, and then it can actually do this concurrently. It's going to run all the pages, and then it doesn't have to run every single page before it starts collecting the data.
So let's go to the next step. In the next step, it's collect info. We're going to start off the exact same by setting the country, going to the URL. This time, we're just waiting until it loads.
Let's look at what one of these pages looks like. The next thing is we want to wait until the page loads before we start parsing information. So I'm just going to find an item that's lower on the page that may load after all the information we're trying to get.
So here we have `react-product-description`. I'm going to wait until that loads. So I'm waiting until something with the ID `react-product-description` loads, and then I'm going to set the timeout for 990,000.
Now, this is overkill; the timeout doesn't really have to be that high, but just to make sure, sometimes it'll have trouble loading, and if I put a really high timeout, then we'll make sure it gets loaded. It probably won't have to wait that long if it finds the element first.
Now, the final thing—there's just one more thing we're going to do—is parse the data. So now we have to go into—we have to write the parser code. Let's figure out what we're going to return from the parser.
Well, we know we want to return the name. So how are we going to get the name? Let's go into here. This is where it can be a little tricky trying to figure out how to get the exact information.
Let me click this item here, and then we can see we have the class `PDP-product-name`. So that's pretty straightforward: `class PDP-product-name.text` will get the text from that element.
Then let's get the brand. We can see the brand here is Bill Jack, and it says `class brand-details`. Next up is the price.
Now, one thing I don't know if you noticed this, but this one has a sale, but they're not all on sale. If I duplicate this tab, we're going to find an item that's not on sale because we want to make sure that the code we write will be able to handle all the different types, all the different ways that the page could look.
So if I look on here, this is `product-price-standard`; that's the class for this one. But here, the class for the price is `product-price-sales`. But look at this; it says data-GTM-price, data-GTM-dprice.
Okay, so whether it's one of them standard or one of them on sale, they both have the data-GTM-price, and then it shows $1.99, which is the price there. So that's how we're going to get this particular element, the price element.
First, we have to select the element with `data-GTM-price`, and then we have to get the value of the attribute of the exact same one. That's how we're going to get the price.
Now, there's another variable thing that I want to show you about. If I go into this one, we can see this has two sizes. So when we get a price, we want to make sure we know which size that price is for.
But not every item has a drop-down menu for the size. So if there's only one size, we just won't even get the size because it's just a single size. But if there's a drop-down menu for the size, we want to get the size.
So if we go over to here, we want to figure out which size, which option is selected here. But if there is no size to even select, we just want the size to be null.
So if we go back over here and look what we have here, we have the size. We're going to look for `size.length`, and if there is a size that length, that means if there is a size, then we're going to get the text of the option that's selected.
Else, we're just going to return null. Then we're also going to get the item ID. It's pretty much the same thing we've been doing. We're just going to see what that is, try to figure out how we can get that.
We're looking for a span that has the item prop of product ID. So let's just add one more line to get the product ID.
Okay, we've actually created each stage, but while you're creating the stage, I should have actually shown how to test it while we were going. But you're definitely going to want to test this out.
So let's go to the URL Setter stage, and we'll just set this. We don't actually need this partial code here. To test it, we just press preview here, and then down here it's going to show you what it's doing.
Now it's finished. It's finished because it's saying that it's actually passed this URL to the next step. If we go to the logs, you can see it did everything from here.
So let's now go to the next step, and for the input, it got this input URL from when we ran the first step, so I don't need this. I'm going to preview this now.
While this is going, oh, this is something very important. It says scroll to function is not supported for worker type. Code should be changed to worker type browser.
If we click the settings right here, we can change the worker type. Now, if you want to simulate an actual browser, which we are doing, we want worker type browser because worker type browser, we are going to actually be navigating through the browser.
So let me try that again. I'm going to go to input and preview, and you can see this is the step. The dot shows what step we're on. So we're now navigating to that URL, and if I click see preview, we can now see the graphical representation.
We can either see the HTML or the graphical representation. You can see it's loading right now. We can't really quite see anything during this when it's previewing. When we're testing it, it goes pretty slow, but once we finish the collector, it actually goes a lot quicker when we're actually running it for real.
But you can see it's loading the page, and it's waiting for the page footer right now. When it times out like that, it could be a sign that you've done something wrong.
Actually, this is not supposed to be page footer; it's supposed to be page-footer. So let's try that again.
Okay, now it went through everything, and sometimes it gets through it so quickly that it doesn't have time to actually show up on here, but it's in the HTML because you can see down here that it got all the URLs and it passed them to the next step.
So now let's test this step. We have this input; we got all the input from the previous one, the previous step, and then I'm just going to—you can see this is a full URL of an actual product, and I'm going to preview this.
This time we'll see if it collects, if it actually gets the data that we're trying to get. I'll see the preview.
Okay, and now it finished going through all the steps, and we can look at the output tab here, the data explorer. So we have the name, the brand, the price, the size—there's no size—and the product ID.
So that's everything we wanted to get, and you can also see the results in a different format. But this was just getting one product, so when we run this, it should be able to get a bunch of products.
So I'm going to click finish editing, and then we are going to eventually try this out. So once you click finish editing, it actually has to test everything.
So it has to test every item one more time just to make sure it's not going to save your collector until it confirms that it can get through all the steps without running into an error.
One thing that sometimes happens just when you're in this mode here is that when you try to preview, it loads slower than it will load when it's in production. Sometimes, because it loads slower, it will be more likely to time out.
If that happens, you just have to click finish editing again, and it's going to run through everything again. You don't necessarily have to make any changes.
Okay, it saved everything; it worked out. So there are a few ways we can actually run this collector. We can run this web scraper. First of all, let's make sure we name it. It's called new collector.
So to name it, we're just going to go back to my scrapers, and the one you just made is always on top. So I'm going to edit collector name and put PetSmart scraper, and then hit okay.
Then I can go back into it, back into the edit the code. I mean, not edit the code; actually, that's not what I want. I want to initiate manually.
There are a few different ways to initiate the scraper manually, where we just start it just like this. I'm actually going to start it now, and then while it's going, I'll show you some of the other ways.
First of all, I'm going to set the number of pages to one so it doesn't scrape every single page. Well, actually, I'll do two. We'll scrape two pages, and then I'll click start.
It's actually going to take a while to run this, so you can see it's giving us some information about what's happening. Right now, nothing has happened; it's found zero pages so far, but we're going to come back to this page.
It's just going to run in the background, and I'm going to show you that.
That's not it; initiate by API. So I went to more, initiate by API. This is how you're likely going to commonly use the web scraper if you're not initiating it manually, through the API.
It's going to give you the curl command. I can actually scroll over, and so we're going to be using both methods. For now, I just want to show you how you get there, and later I'll show you how it works.
You can also go to subscription and set it to run on a schedule. Basically, you can say when it's going to start, how often it's going to run, run until finished.
This is something that you may want to set up, and then for delivery preferences, you're going to say how do you want the information to come to you in JSON, CSV, and so on.
Then what's your delivery strategy? Is it going to be emailed to you, or are you going to get it from the API download, or you can have it delivered straight to Amazon S3, a webhook, or all sorts of places you can get the data to go wherever you want it to go.
So I'm just going to keep it on API download, and I think that's one of the simplest ways. You can see a lot of other settings that you can set up here, but I'm going to go back over to runs, and we're just going to wait until this finishes.
Okay, this finished. This one actually took 10 minutes to finish, and remember, it's usually going to be running in the background. So the amount of time it takes isn't really that important because you'll usually have it set up to run automatically.
You can see it went through 75 pages; it got 72 records, and here we can download the file. So I'll just click on download JSON, and here I can see the JSON file.
So we have name, brand, price, product ID, and then it also gives the input that we put in. Now we have all the information that we got from two pages of products or two pages of search results, and then every product has its own page.
So now let's look at running it through the API. First, I'm going to put in that we're going to run it one time, and it's going to give us this information here. I'm just going to copy to the clipboard, and then I'm just going to go back to some text editor and paste it in right here because we're going to have to update some things.
Specifically, I'm going to have to put in the API token. So to get an API token, I'm going to go to settings, account settings, and then I can go to the bottom here and create a token.
I already have a token, but just so you can see how it works, I'll revoke the token, and then I can add a token to create a new token. Then I click save, and it's going to give this. This is the only time you get to see the API key, and don't worry, I'll revoke this before this video goes live.
So I'm going to copy this key, got it, and then I'll just paste it in right here. So I'll take this whole curl command and paste the curl command in my terminal here, and you can see it sends back this collection ID, which is this, and the start estimated ETA.
So it's now started running the scraper. If I go back over to this, let me go to my scrapers, and then actually I'm going to go to runs here, and you can see it's actually running right now. It's running via the API.
If I go back over here to initiate via API, we can see how we can—there's also a results API. So to get the results of running the API, I'm just going to copy, even though it's grayed out, I can still copy this and bring it over to my text editor.
It's just this is the new curl command, so I just have to replace the collection ID with this collection ID I get here, and so I will paste that in.
Once the job is done, we'll be able to get the information back, and I also need to get this API key, and I'm going to paste it right here. Even though the job is still going, I can run this curl command.
So if I run the command now, it's going to say status collecting job is not finished. So I just have to wait until the job is finished. Now, I could keep going to the API to see if the job is finished, or I can look in the web interface.
Okay, we can see it's done on the website, so let me just run that curl command again.
Okay, the data is not ready; try again in 10 seconds. And here we go; we can see all the data right from the curl command.
We've seen how to initiate the scraper from the API and how to get the results. Now let's see how we can automatically update the cost of our demo store here based on the prices of PetSmart so we can make sure our store stays competitive.
So let's see how this works. It's just a simple React app created with Next.js, and we actually get all the information. All the information is stored on MongoDB; it's stored on MongoDB Atlas.
So we get all the information from this API endpoint that has all the information about the products on our Pet Brigh store, and you can see it's going to return just these cards. It's just going to return the cards, and it's going to have information like the price, the image, the name of the brand.
This is all coming right from our database, so if we update the database automatically, the price will update on our page. I created this database on MongoDB Atlas. It's pretty simple to create a MongoDB Atlas account; it's all free to create an account.
If I click browse collections, we can see the database collection. You can see here we have Pet Brigh, and it's the Pet Brigh products. You can see here all the information with the name, the brand, the price, the product ID, and the image link.
The way I made that API endpoint to connect and get all this information is in App Services, MongoDB Atlas App Services. In App Services, I have this HTTPS endpoint, and I have this products endpoint.
Basically, I just had to click add an endpoint here, and then I called the products endpoint, and then I had to make it a GET request, and then I set it to return JSON.
For a function, I did to create a new function, but I already have this products function. If I go to the functions here, we can look at the products function. We can see that this function is returning the information from the MongoDB Atlas database.
So the database is Pet Brigh; the collection is products. We're finding all of the records, and we're just returning 20, just to make it simple. We're just returning 20 items from the database, and then I had to go through and make sure the authentication was set to system.
That means that anybody is going to be able to access this API endpoint. If I go back to the API endpoint, we can see the URL here, and this is just the exact URL that is in the React app.
So I'm going to create a new function that will update our database using the information we scraped from the PetSmart website. So I'm going to go to functions and create a new function.
Now, there are plenty of other ways you can create cloud functions. You don't have to do it through MongoDB Atlas. If you know about AWS functions or functions on any other cloud-based service, but I'm going to call this function update prices.
Again, it's going to be system, and then I can go to the function editor. This is just all example code, so I'm going to get rid of all this example code, and I'm just going to quickly go through how to make this function.
First, we're not going to care about the query headers and body now, but if we ever wanted to update it, it's good to have this in here.
First, we have to get access to our collection, so it's the Pet Brigh database and the products collection. Then I'll just paste in the rest of the code, and then we can go over it pretty quickly here.
We're going to first make the API call, so we're going to go to this URL. So that's the URL right here, and then we have to authorize the bearer with our API key that's right here.
Then we're just going to parse the response. We're going to get the API response, and then we're going to do `JSON.parse` to get this into a JSON object, which is basically just an array that we're going to go through.
So for each product in this array, first, if there's no product ID, we're going to skip it. But if the element does have a product ID, then we just call `collection.updateOne`. That's how we update just a single item in the collection.
We're going to find the item that has the product ID that we got from the API, and then we're going to set the price to be the product price. So we're basically updating all of our prices so they are the same as the prices on PetSmart.
Now, we could also make it so maybe it's like 5% lower, 10% lower if we wanted to make it even lower, but we'll just try to keep them the same price. Then we'll return a message: products updated successfully.
So let's actually load in the prices that we scraped from PetSmart and see what happens. There are a few ways we could run this function. First, I'm going to save it, and I'm going to review draft and deploy.
So we've deployed that function. We could set up a trigger so it's going to run at certain times. We could schedule it to run at certain times. We can set up an HTTPS endpoint so the function will run whenever it receives a GET request to that endpoint.
I think it would be better to have a trigger. So if we click add a trigger, we could set the trigger to be scheduled, and we can set when it's going to run.
So we could actually set it to run every day at 2 a.m., and we would select the function. We would select the update prices function, and then we could go back to our Bright Data and run the scraper every day at 1:00 a.m.
So if we scraped at 1:00 a.m., that would give it time to finish by 2:00 a.m., and then at 2:00 a.m., we would run the function that would update our prices. But we can also just run it manually, so let me just show you how we're going to run it manually.
I'm not even going to bother setting up a trigger right now; I just wanted to show you how you would do it. I'm just going to go down here and click run.
Oh, I didn't need this line here. Okay, now I'm just going to save this draft again, view draft, and deploy, and then I'll run it. Products updated successfully.
Okay, so we remember that the cost of this product was $81.99, but for us, it was $84.49. I haven't refreshed it since running the function, but I'm about to refresh the page.
We're going to get the updated prices. So the old prices are $84.49, $92.48, and we'll refresh. $81.99, $89.98. So $81.99, that's what the cost is at PetSmart.
So we've done it! We've successfully scraped the prices from PetSmart, and then we wrote a function that will update our database with the prices that we scraped from PetSmart.
That will help us stay competitive, and obviously, you can use this for any type of web data.
Well, thanks for watching! You can now put "web scraping expert" on your resume.