Transcription
One of the biggest challenges we face with large language models right now is that their knowledge is too general and very limited for new things because of their training cut-off. For example, my favorite new shiny framework for building AI agents is Pantic AI. But if I go over to Claude right now and ask it what Pantic AI is, it has no clue. Even if I use an LLM that can search the web, the information I get back is still going to be very bare bones.
On the other hand, if I take all of the framework documentation for Pantic AI and put it in a knowledge base for the LLM, and I ask it the exact same question, the answer I get is spot on. That is why RAG is such a huge topic when it comes to AI right now, which, by the way, stands for retrieval augmented generation. It's a method for giving external knowledge that you curate yourself into an LLM, basically to make it an expert at something that it wasn't before, like an AI agent framework, your e-commerce store, you name it.
The problem is that curation can be very difficult and slow. For example, if you want to ingest an entire website into a knowledge base for your LLM, how do you actually do that and get it done fast so that it's not 2027 by the time your knowledge base is ready and AI has taken over the world? Anyway, that is where Crawl for AI comes in.
Crawl for AI is an open-source web crawling framework specifically designed to scrape websites and format the output in the best way for LLMs to understand. The best part is it solves a lot of problems that we typically see with these systems for website scraping. Usually, they're very slow, overly complicated, and super resource-intensive. But Crawl for AI is the complete opposite. It's super intuitive, very fast, easy to set up, and extremely memory efficient.
In this video, I'll show you how to super easily use Crawl for AI to scrape any website for an LLM in just seconds. At the end of this video, I'll even quickly showcase a RAG AI agent that I built, basically to be an expert at the Pantic AI framework, of course using Crawl for AI to curate all the framework knowledge into my knowledge base. Really, you could take what I built and use it for any website. Super exciting stuff!
So let's go ahead and dive right into it. The two big things that we're going to focus on right now are Crawl for AI, which this is their GitHub right here, completely open source, and then also Pantic AI. Now, this is not a Pantic AI video, but it's just a very good example of full documentation that we can scrape with Crawl for AI and bring into a knowledge base for our LLM.
Now back over to Crawl for AI. The first obvious question when we visit this page is, what is the point of Crawl for AI? Why even use it? Luckily, at the top of their README here on their homepage, they have a very concise description for why you should care about Crawl for AI.
The first big thing is when you visit a website and extract the raw HTML from it, it looks like a mess. As a human looking at this, it's very hard for us to actually extract useful information from it. A good general rule of thumb is if it's hard for us as humans to understand something, it's probably harder for a large language model as well.
One of the most important things that Crawl for AI does is it takes this ugly HTML that we get when we visit the raw content of a web page and turns it into markdown form, which is actually a human-readable format. It's a super clean way to represent a web page and all the information that we see when we typically visit a UI, just in something that's all text-based. So we can give it to a large language model for RAG and just for LLMs to understand it better in general.
That's the first thing, and it does it very efficiently. It's super fast and handles a ton of things under the hood, like proxies and session management—things that are not easy to handle. So it's not like you can go and make your own version of Crawl for AI very easily just because you know how to use the request module in Python to pull HTML from websites.
It's also completely open-source and very easy to deploy. They even have a Docker option, which I'll probably cover in another video on my channel later on. They're doing major updates soon to their Docker deployment, so I don't want to focus on that now, but just know that that is available as well.
There are a ton of other things that they do that are super valuable as well. One really important thing is removing irrelevant content. I mean, when we go to the HTML here, we have a ton of script tags, and there's probably a lot of information that we can view on the page that is very obviously redundant or just not useful to us. They take care of removing that as well.
So eventually, what we get back from scraping the site with Crawl for AI contains just what we care about actually ingesting into our knowledge base. Getting started with Crawl for AI is so easy. All you have to do is PIP install the Python package and then run this setup command, which is going to install Playwright under the hood. That is the tool that Crawl for AI uses to scrape websites and basically have this browser running in the background that can visit these sites.
Playwright is a fantastic, also open-source tool that I use for a lot of testing for my web applications as well, so I'm very familiar with it and can recommend it. I think it's a great choice for actually having that web scraping functionality under the hood. Getting started is so easy; it's just as simple as this little script right here.
I actually have my own version of it. If I go into my source control here, I've got my own version of it just scraping the homepage of Pantic AI, their documentation. So let's go ahead and test this out and see what kind of output we get. By the way, all of the code that I go over in this video I'll have in a GitHub repository that I will link below.
Right now, we're starting with just these basic examples, starting with this one to pull the homepage of the documentation for Pantic AI, and then we'll get to my RAG AI agent later, and I'll showcase that a little bit as well. I've already installed Crawl for AI; it took like 30 seconds to install everything, including Playwright under the hood.
Now I can just go ahead and run our script, and within seconds, we're going to have the entire page printed out in the terminal here. So fast! I'm very impressed with this. This isn't like the perfect format for us as humans to understand because we have all this little markdown syntax and stuff, but this is definitely a great format for an LLM to understand this entire page, especially compared to what we get if I just go back to the Pantic documentation here.
I'll just open this back up. If I inspect the page source, this is the HTML that we got the markdown from. Imagine pasting all of this into an LLM prompt. I mean, it's just a mess. It's definitely going to hallucinate when you try to ask a question with all of the HTML tags and everything in there. So it's just so much better when we have something that looks like this.
All right, so we saw a basic example using Crawl for AI to get the markdown for a single file, but obviously, we have to take this a lot further to do something that is actually useful for our LLMs. The first thing that we want to do is make it possible to ingest every single one of the pages in the Pantic AI documentation.
We want to get the markdown for the introduction page, installation, getting help, contributing—all of these at the same time. The first problem we have to tackle to actually make that possible is we need an efficient and scalable way to extract all of the URLs from the documentation here.
I could just go and manually copy and paste the homepage and the installation page and the agent page and just bring that into a list in my code, but that is very inefficient and not scalable. As more pages are added, I'll have to manually do the same thing and constantly update the list myself.
Luckily, there is a very good solution for this: introducing the idea of a sitemap. For most pages out there on the internet right now, if you go to the homepage and then add /sitemap.xml, it's going to give you this XML, which gives you the entire structure of the website—all of the pages that exist there.
You can do this right now for the Pantic documentation, like I'm doing right here, and all the pages that you see here are the same ones that we see in the navigation. I can do the exact same thing for the Crawl for AI documentation. It's very meta, but I could use the sitemap.xml to get all the pages in the Crawl for AI documentation to scrape with Crawl for AI.
You can do this with most websites. Most e-commerce stores, like if they're built with Shopify or WordPress, will have this as well. In general, websites have this for search engine optimization and also for crawlers. A lot of websites want you to crawl them because they want their information very widespread.
If I'm building an AI agent framework, I want people to crawl this and build agents around it because then they're using my framework, and it's just more accessible. By the way, if you are curious if you can scrape a website, there's a lot of ethics behind web scraping. Typically, what you can do is go to a website like youtube.com and then add /robots.txt to the URL. This will give you a page that tells you their rules for web scraping.
This says that any agent is allowed to scrape YouTube; however, there are certain pages that are not allowed. This is super important to keep in mind if you want to be very ethical with your web scraping, which I highly recommend. Check the websites you're scraping for robots.txt first before you just go ahead and do it. GitHub is another good example here, where they actually say if you want to crawl GitHub, please contact them first. A lot of them will be like this, so keep this in mind.
I very much owe it to you to provide this little segment talking about ethics before I dive into the rest of the video. It's very fitting to do that because we're talking about URLs that you can add to pretty much any website. You can do /robots.txt or /sitemap.xml.
We're going to code to pull this sitemap, extract every single URL, and then feed all of those into Crawl for AI. We want to do that very efficiently as well because right here we're just going to be pulling in every URL and then looping and going through one at a time. If we do it just in a loop with this code right here, we want something more efficient.
If we go to the documentation for Crawl for AI, which is right here, that'll bring us to this page. If we go down to multi-URL crawling, there is a lot that Crawl for AI gives us for this, and that's what we're going to be leveraging for the rest of this video.
First of all, if you just crawl your websites in a loop like this, like we would do if we just continued off of this example right here, it would be very inefficient. We're spinning up a brand new browser for every URL that we are visiting, and there's no opportunity for parallel processing, which we're going to get into as well.
Their recommendation gives a full example for how you can use the same browser session for all of the pages that you're visiting and pulling. That brings us to the second script that I have built for us here. I'm not going to go over all the code in detail because this is mostly following the example that we just saw right here.
I just copied this in, brought it into my code editor, and then I have this custom function right here where I pull that sitemap that I just showed you. I pull it, extract using XML processing all the URLs from the sitemap, and then I pass them all into this function to crawl the URL sequentially with the same browser session.
The code gets a little complicated with the browser config and crawler config, but don't worry about that. In general, you can just take this example and use it for yourself. It crawls every single URL, and it's not going to print out the content of every markdown. That would just be way too much in the terminal. It'll just show us the length and whether or not it succeeded in crawling the site.
I'm going to go back to my terminal here and run this second script. It's going to take a little bit because it has to crawl all of them sequentially. We'll get into parallel processing next to make this even faster, but even this just took seconds. It was so fast processing each one of these pages, giving me the length and whether it succeeded or not for each one of these URLs.
At this point, we already have a very fast way to get the markdown for every single Pantic AI documentation page, and it's ready now for us to put in a vector database for RAG to use with our large language model. It's super neat, and it was so easy to set this up with Crawl for AI.
But before we actually get into anything with RAG, I want to take this one step further because I want to make this even faster. It was already fast here, but we're still processing each one of these URLs sequentially. There's no parallel processing, and we can definitely do that with Crawl for AI.
We can visit multiple pages at the exact same time, pull the markdown for every single one of them, and then combine it all at the end into a single list, just like we're doing right here. The way that we can do that, if I go back to the Crawl for AI documentation and just scroll down a little bit, they have an example doing exactly this—parallel processing.
It's essentially going to be the same; we're still just using one browser, but we're creating different sessions that are all going to be up at the same time, visiting these URLs in parallel. Just like last time, I mostly just copied the example that they had right here and then brought it into my code editor.
Again, just like last time, the main thing that I added is that function to use the Pantic AI sitemap.xml to get all the pages that I want to pull the markdown for and scrape. I'm going to open up my terminal again, and actually, one last thing before I do that: for the batch size, we are doing 10.
It's going to visit 10 pages at the exact same time, get the markdown for all of them, and then move on to the next set of 10, and then repeat that until it has pulled the markdown for every single page. I have a new terminal open up right here. I'm just going to run this script just like I did before.
Last time, I showed how fast each run was; now I'm going to show how memory efficient it is. When I run this, it'll show the current RAM usage for the script, which starts at 91 megabytes. It's going through all these batches very quickly, and at the end, we can see the peak usage, which is only 119 megabytes.
Throughout this entire time, even though there's an entire browser running in the background visiting 10 pages at a time, it's still only ever used 119 megabytes of memory at once, which is just incredible. The last example was actually basically as fast, but that's only because of caching. In general, this batch processing is going to speed it up a ton, which is super impressive.
Now we have the perfect thing for RAG because we're doing it very quickly. A lot of times, you need that. This example has 42 pages, but if you have something like an e-commerce store with hundreds or thousands of products, you can imagine that this is going to start to be a drag if you are processing things sequentially, not using the same browser and same session.
That's why using Crawl for AI and having all these efficiencies is so important. The very last thing, and this is my true gift to you: I have already built out a full RAG AI agent that is a Pantic AI expert. Using the exact same process with Crawl for AI that we just did, I pulled all of the Pantic AI documentation and then put it into a vector database for its knowledge base.
I built a full agent around it and created a front end that we're looking at right here. My gift to you is this is already available to you. I have the code in a GitHub repository that I have linked below. In the next video on my channel, I'll be covering how I actually built this agent, and it will be available on the live agent studio for you to try immediately.
Super neat! Also, a little bit of a sneak peek right now. Let me paste this in here. I'll just ask a basic question like, "What are the supported models?" This is the kind of thing that Claude or any other general LLM would definitely not have the answer for, and it even links me to the different pages in the Pantic AI documentation for my reference.
Very neat! I can ask a ton of other questions as well. For example, I can say something like, "Give me the weather agent example from the documentation." Obviously, I just know that this agent example exists, so I'll ask for it, and it'll go and search for it, find this full example for me, and it does it so fast as well.
This is perfect! This is a pretty complex example because it's showing me basically every part of creating an agent with Pantic AI, which is super neat. So there we go! This is the full agent, and I'll be showing you exactly how to build it very soon on my channel.
The reason that I'm not covering how to build the entire agent in this video is that I want to keep it concise and focused on just Crawl for AI, especially because there are a lot of other use cases for web scraping besides just RAG, even though RAG is definitely one of the biggest ones for AI right now.
If you go to the GitHub repo, I have a README right here covering everything with this agent. You have all the code for my entire process of crawling all these sites again with a very similar process to what we went over in this video, and then actually inserting that into our vector database, which I'm using PG Vector with Supabase here.
I have my agent that I built with Pantic AI—very meta, but it is my favorite framework right now—and then I have my Streamlit interface. All this is available for you with instructions on how to run it yourself. Stay tuned for my video later this week, where I'll show you exactly how I built it.
So there you have it: a bulletproof, lightning-fast way to scrape any site and give it to your LLM as a knowledge base. This is useful for you pretty much no matter what your use case is because there is almost always a time and place to take data from external websites and bring that into your LLM.
In my mind, this makes Crawl for AI a game changer. Don't get me wrong; there are a lot of ways to bring knowledge into an LLM. You can manually curate data, use new advanced concepts like KAG, and a lot of things I'll cover in more videos on my channel.
But it is still the most common way to make an AI agent an expert at something you care about: to scrape data from a site and provide it to a knowledge base for RAG. In the next video on my channel, I'll do a deep dive into the RAG AI agent that I demoed earlier, which I'm super excited about because I put a lot of effort into building it for you.
If you appreciated this content, I would really appreciate a like and a subscribe. With that, I will see you in the next video.