Transcription
Today, I'm going to show you guys how you can take any website and turn it into LLM ready data in seconds. We're going to be able to take any website and fully scrape everything from it. We can get all the screenshots, the branding, we can map out the entire site, we can extract the data in any form we want. We can pretty much do it all. And we're going to be doing all of this through cloud code using a tool called firecrawl, which I'm sure you guys might have heard of before.
So, within Firecrawl, there's a lot of different things that we can do. We can scrape content, so get everything from the page. We can map out a site, so get all the different URLs and understand the architecture of that website. We can crawl it, so then explore all of those pages. We can actually search, so do like a web search first and then scrape the data, which means that we can turn pages into structured content for us to use however we want. Now, the thing about this is there are a lot of different endpoints to hit if you were doing this through traditional like API calls. So, we're going to be using the MCP server for firecrawl. We're gonna give that to Claude Code so it can figure out, based on Nate's natural language request, which of these tools do I invoke and in which order do I use them to actually get the end result that he's looking for?
So, just to start off with a quick example, I'm going to use Firecrawl in the playground, which is just on the web on firecrawl.dev. You guys can get here using the link in the description. You can get on the free plan, which is more than enough to just play around, and then when you're ready to upgrade, use the link in the description and you can get 10% off a plan. But anyways, I'm going to go to Upai and I'm going to copy this URL, paste that in here, and I'm going to run a scrape. But first, we have to choose the format. So, by default, it'll pretty much say, "Okay, we're going to turn this website into markdown for you." But what you could also do is get an AI generated summary. You get all the links. You can get HTML. You could get a screenshot of the full page. You could get the branding scraped so you can understand like the logo and the images and stuff like that. So, I'm going to go ahead and start the scrape. And what we'll see is it'll pop up down below. And we'll get all of this data.
So, that just finished up. We can see, first of all, we have markdown. So, we get the actual like hero text. We get all of this stuff up here. We've got the process. We have the testimonials, get in touch, all of this kind of stuff. We have an AI generated summary of what the business aims to do. We've got a screenshot of the entire page. As you can see, we've got branding information. So, the OG image, the favicon, the logo, colors, typography, stuff like that. And then we also have the JSON, if for some reason you're crazy enough to want to read this.
So, like I said, we're going to get this into cloud code. So, I'm going to click on the docs. And on the right hand side, or on the lefth hand side, we can see MCP server right here. And then what we want to look for is running this on cloud code, as you can see right here. And now this gives us basically just this one line to put into cloud code, and it will be able to install this for us. So, I'm just going to go ahead and copy this. I'm going to go into VS Code, and we're going to start up a new project and get everything initialized. So, if you've never actually been in VS Code or worked with cloud code, then I'm going to link a video right up here. It'll get you caught up, and then you can come back over here when you are ready.
So, what I'm going to do is open up a folder. So, I'm just going to open up a project called scraper. There's nothing in here. It's a completely new project. So, this is what your guys' setup should look like. Left-hand side, nothing. Right hand side, open up cloud code. And I'm just going to say, "Hey, Claude, I want to connect to Firecrawl's MCP server." And you can do that using this command. But I'm not going to give you my API key. I'm just going to put it in av. So, if you could create that file for me, I will put my API key in there. And then you can go ahead and initialize and connect to Firecrawl's MCP server.
All right, so that's going to go ahead and get set up for us. You can see we now have a ENV right here, which is a new file. And this is basically the best way for us to securely put in our API keys so that they're not being stored in like the conversation history. So, I'm going to delete this. I'm going to go back into Firecrawl. I'm going to go to my dashboard. And right here, you can see there is an MCP integration, but we wanted to use the cloud code version. And now we can see API key. I'm going to go ahead and copy that, paste in the API key right there. And then I'm going to do CRL S just to save it. You could also go here and click file and then save right there. And now we can close out of that file. My API key has been added to ENV. Go ahead and set up the firecraw mcb server.
So, now everything should have been set up correctly. So, I'm going to hit control shiftp and I'm going to say, "developer reload the window," which is just going to actually let cloud code be able to use this now. So, just sending off a request to make sure that that actually worked. As you can see, it was able to call the tool, and it decided to use the scrape endpoint rather than, you know, like a map or a crawl. So, this is the actual like full markdown of the website itself. Cool.
So, the next thing I want to do is give our project a little bit more context as to what it's actually doing in here. So, first of all, what I want to do is create a fire crawl MCP guide so that when we ask it something, it understands what are the tools I have access to and which ones should I use for what scenario. So, I said, "create a firecol cheat sheet as a markdown file in this project that you can look at." And basically, it should tell you about the different tools and how to use them. So, it's going to go ahead and create this markdown file for us.
All right, cool. So, it created that cheat sheet, as you can see, right here. And if I just open up this full screen, you can see that we've got a quick reference guide. We've got the tool overview. And then it goes on to actually break down how you use each tool. So, this should hopefully be good enough for now Claude to look at whenever we want it to do something else. It even gave it a quick little decision guide, which is pretty nice.
Now, finally, before we actually start running with this thing, is we need a claw.md file, which is basically the system prompts for this project. "Hey, Claude, I need you to help me set up a claw.md file for this project. I want this to basically explain that this project specifically is for scraping data. Whether that is extracting it, getting screenshots, crawling everything, mapping everything. You have access to the fire crawl MCP server to do everything that you need to do with websites. And you also have the firecrol-cheatsheet.mmd, which explains how that MCP server works and when to use each tool."
So, I just shut that off. Now, I just did want to say this is a demo, right? So, I'm doing this all in bypass permissions mode. But in practice, what I would have done is went to plan mode, brainstormed with claude a little bit to make sure that it agrees with like the way that we're setting up all these files, and then we'd go ahead and implement that plan once we are in alignment. But, as you can see, we now have our claw.md file, and this is basically a scraper project. We have some information about what this project does, how to actually use the tool, what to reference. And this document, as you can see, is a lot more concise than the cheat sheet. And the reason I wanted to separate this is because you don't actually need this entire cheat sheet to be in the cloud.mmd file, but now claude knows that it's there in case it ever needs to use it.
Okay, so let's think about a cool use case that we might actually want to do with something like firecrawl. So, let's say we've got this remote job website, and I search for content, and there's about 1,700 different job opportunities here. And there's also, I'm assuming, not all just on one page. So, there's two, three, four, maybe even up to 60. So, I'm just going to go ahead and copy the URL of this first page right here. We're going to go into Claude and ask it to help us out with this. "Hey, Claude. So, I found this website, and I've got about 1,700 job opportunities that I want to look at. But I need help using the Fireall MCP server in order to get all of these listed out. I want these as structured data, so I could maybe just throw them into a Google sheet."
Now, in this case, I am going to go to plan mode because this might take a little bit of thinking as far as understanding the structure of the site and maybe using more than just a scrape. It might have to use a map, or a crawl, or something else. So, we'll see what it decides to do here. So, hope you guys see now why I did this. It first decided to scrape. It understood the website, and then it decided to map. And now it's creating more of a comprehensive plan about what to find. It's also asking me some questions, which is going to make this job work a lot better.
So, it asks if we want all 1782. I'm actually just going to go ahead and say like 200 because I don't want this to take forever. For data fields, I'm going to grab all of them. For description, let's just do a summary, and I'm going to go ahead and submit those answers, and it's going to keep working on the plan.
All right, so looks like that is all done. We're going to go ahead and autoaccept this plan. So, I'll check in with you guys when that's done. Now, right here is the beauty of Agentic Workflows, because it tried to, you know, execute the plan, but once it got into it, it realized that something didn't work. So, it said that the extract actually returned empty results, and the site might require more sophisticated handling. So, now it's trying out the firewall agent. So, just super cool the way that it's able to, you know, run into an issue and then fix it.
Okay, so that just finished up, and it was able to get 200 job listings for us. A few things happened in there, but it was able to just correct itself and change up the plan, and we did get our final output. So, let me open up this CSV. You can also see that it dropped it in this project, and we could look at it over here, but it's not really very nice to look at. So, here's the actual Excel file. We've got title, company, job type, location, salary, experience, category, posted, how long ago, apply URL, description, and tags. And we do indeed have 200 of these. So, now, if we wanted to apply to all of these, we've got all the URLs, and we have all this info that we need in order to go and do that. So, think about how long that would have taken you to build an automation in something like Nitn in order to go scrape 200 of these job postings, or if you were to just do this manually, it would have taken a lot longer. And once again, I didn't have to think about any of the configuration. I just gave Claude Code the MCP server and let it run.
We're going to do two more really quick use cases. The first one I'm going to do is grab Cloudbot or Moltbot and drop it in here and say, "Please grab screenshots of this page and help me understand the branding." And I'm assuming that this is going to use the Firecol MCP server. I hope it does. And then I'm going to grab this website, which is coffee, and I'm going to open up a different agent on the right hand side. And for this one, I'm just going to tell it to map this site. Go ahead and map out this site for me and show me what it looks like. So, now we've got two different Cloud Code agents working at the same time. They're both doing different tasks, and they're using different firecrawl tools. And then I'll check in with you guys when we get both of these back.
All right, so the map is already done. You can see that it comes back and it says, "Okay, so here are all the main pages." And it gives me the links. Here are all the different categories. So, best sellers, coffee, instant, matcha, all this other stuff. We've got different collections. We've got different locations. We've got tons of different URLs for products, brew guides, all this kind of stuff. And so, now that it has that context, I could have it go actually crawl those things if I wanted to, or, you know, extract all of that to a database, or whatever we want to do with it. And now it looks like this one is finishing up over here with the Moltbot documentation. So, the first thing we have is a screenshot. So, if I open this up, we can see right here that we do have a screenshot of that whole landing page for Moldbot. And then we also have the branding, like the color palette, the typography, spacing, and components. We've got the logo. And you can see that all of this was able to be done with Firecrawl super easily.
So, I wanted to show you guys all of that stuff that we just did together, what that actually costed me. So, I'm going to refresh my dashboard here when we're looking at the usage. And you can see that that took me about 30 credits out of my 500 that I get for free. So, 6% of my 500 credit limit. Now, that's really the main difference when it comes to pricing. You've got these different plans. You've got a different amount of pages that you can scrape. You got a different amount of credits. But the other big one is the concurrent requests. So, with the free plan, you can only be doing two at a time. With this hobby plan in the middle, you can be doing five. If you scale that up, you can be doing more and more. And it's not really a huge deal because what would happen is Claude Code would basically just cue them up and wait and retry. But if you did want to do some big operations in bulk, then it may be nice to have more concurrent requests running. And remember, you can use the link in the description to get 10% off your Firecrawl plan.
So, I know this one was quick, but I really thought that it was a useful tool and skill to add to your Cloud Code workflows and projects. So, I hope that you guys enjoyed. If you enjoyed this teaching style and this way of thinking, then definitely check out my plus community. The link for that will be down in the description. We've got over 3,000 members who are building businesses with AI every single day. So, it's a great place to be if you have similar goals. We've also got a classroom section with full courses, and we're bringing a ton of more material around cloud code and vibe coding. So, if you guys are interested, then definitely check it out. But anyways, that's going to do it for today. If you enjoyed the video or you learned something new, please give it a like. It definitely helps me out a ton. And as always, I appreciate you guys making it to the end of the video. I'll see you on the next one.