Transcription
Back in my agency days as an SEO specialist, a huge part of my job was working with data, competitor research, PR reports, backlink building, and it meant scraping a lot of information from a lot of websites. The problem was it was either crazy expensive to do, crazy complicated, or a massive manual time suck. And I spent most [clears throat] of the time actually trying to get the data, not work with it and deliver results.
But that's all changed now because you can do all of this conversationally. No code, no technical knowledge. Just tell Claude what you want and it'll go and get it. So in this video, I'm going to show you exactly step by step how to use Claude Co-work to scrape any website on the internet. Let's jump straight in.
Okay, so first thing we need to do, we need to open up Claude Co-work here. You can see this is on the desktop version of Claude. So if you don't have the desktop version, go and download it, come back and start. Then we want to hit the toggle in the middle and move on to co-work because this is where it's going to work best within Claude. It's going to be able to run for much longer and work with much more data.
Now, to activate its scraping mode, we actually need to press the plus button in the middle here and go to connectors. You can see I've got a few running here, but I'm actually going to show you go manage connectors. And this is where you can get a better look at the sort of connectors we're going to use. The main one I'm going to show you how to use today is Appify.
So, the first thing you need to do is go and add this connector to your co-work. To do that, hit the plus button at the top here, go browse connectors, and you can just search for it. You can go API, and you'll see it comes up here. Click on it and hit the install button there, and it'll install that connector for you.
Now, you'll notice here that I've got my Ampify token. This is the API key that's actually going to allow Claude to connect to this service. So, the other thing you need to do is you need to go and set up an Apeify account. Apy or Apify? I'm not sure. I keep jumping between the two.
So in your web browser go to apify.com and this is the homepage. This is basically a marketplace for thousands, tens of thousands of different website scrapers. You can see here it can scrape Tik Tok, Google Maps, Facebook post, Amazon website content crawler. It can go and crawl any website. So this is insanely powerful.
Now you want to set up an account here. And once you've set up an account, you'll get your console open like this. And you can see I've been using it to scrape Facebook ads a lot of the time or browse websites to pull information from them. Now, the beauty of Apify is you get a free $5 limit every month. And I use it, you know, every now and then, and I've only spent a $1.15 over the last month. Honestly, it's less than a cent per scrape result, basically. You can see this one, for example. I think I scraped a few dozen Facebook ads. It cost me just over two cents to actually run. So, it's actually really cheap to run.
Now, once you've set up Apify, you'll see the create button in the top right of the screen here. If we hit that, we can say create a new token. And these tokens provide API access to Claude or anything else you want. So, you could say Claude co-work in here so you know what it is. You could have an expiration date, the amount of limits you want to put in there, but then just hit create and it's going to give you your API key to then use. So, you can then copy this API key. Keep it secret. You don't want to show it to anyone. You can see here. I don't want to show you this cuz I don't want to give anyone else connection to it. Copy that token. Take it and put it into your Apify little token area here on co-work. And then you're ready to rock. That's connected. You're all good to go. There is an update for this as well. So, I'm actually going to update this connection as well before we actually run it to get the best results.
So, with that all set up, we are ready to go. We can go back to our co-work chat here and start prompting away. Now, I'm just going to jump into this is my AI recipe vault. And I've got all the step by steps of how to do this saved in here. And there's a link in the description where you can get all of this. It's got all the prompts and everything I use in this video and all this set up. I'm actually going to go and grab the prompt that I've got in here to use in this video that you can use as well. And take it back over into co-work.
Now, you can see it's very simple. It doesn't need anything complicated. Use a proifier to scrape 20 solar panel panel installation companies in the southwest of England. For each firm, collect the following details: business name, website, contact person, email address, and put everything into a clean spreadsheet. That's all I need to say.
Now, there are thousands of scrapers on Apify. I don't even have to specify which one. I can just give it a job. I've not even said where I want it to scrape it from. I'm going to let it all work that out itself. If it should go to Google Maps, if it should go to some directory. I'm going to leave Claude to work this all out itself in tandem with Apify. And then you'll see it's gone and searched for the web scraping tools in Apify. It's loaded them and it's decided I'm going to scrape Google Maps for solar panel companies and then run it. So, it wants to ask for permission. I could say allow once. I could say always allow. For now, I'm just going to say always allow. And after a few minutes, you can see it's worked its way through the problem here, found different areas, different businesses, and put them into our spreadsheet. So, we can actually open this spreadsheet in the chat here or actually in Microsoft Excel, which is awesome. You can see here we've got our list of different companies everywhere. We've got the website. We've got email address. which we actually weren't able to find, but I'm going to show you how we can go and get that in a minute. We've got phone numbers we can ring. We've got locations and much more in there. So, instead of either having to track down this information yourself and put it in a spreadsheet or work out how to technically go about scraping a website, you just ask co-work to do it and it goes off and does it for you, puts it in a spreadsheet, which we can keep updated and adding to.
So, let's say we want to move on. We don't actually have all the contact information about people and email addresses and that sort of thing. So, it's really hard if we want to do outreach to them to actually do this without only cold calling. We don't even have a name to actually know who we want to speak to. Well, we can add another connector which can help solve this problem for us. Let's go back to manage connectors here. I'm going to add a fresh one to show you how to set it up as well. This one's called Vibe Prospecting. And this enables you to get even more information easily, emails, phone numbers, and help build out some more qualified leads.
So, I'm going to install this. And while that installs, I'll go over and show you this platform. Now, you'll see you can actually use this directly on the web app itself. But when we're pairing it with Claude Co, we get even more power. So, when you log in with a free account into their app, you actually get 400 free credits to start using this. So, again, we can start using all of this for free to test it out. Even more excitingly, at the top here, Vibe Prospecting runs best on Claude. And when you hit that claw button, it'll give you the download and everything like that. But we can connect this to Claude and start running it on full power with Co-work. As you can see here, it's installed. It did like an authorization to connect the accounts. It was super easy. Don't even need API keys and it's ready to rock and roll.
So, let's go back to our chat here and start collecting even more information. Now, back in our document here, I've got the steps for the Vibe prospecting as well and another prompt that we want to use that will take over. And we will pop that into claude at the bottom here. And I said, you know, take the list of solar panels we just scraped. Use vibe prospecting. I might just put connector in there just to be even more clear to find key decision makers, founders, directors, managing directors. I just want to tidy up the prompt there a little bit actually. Find a key decision maker name, job title, LinkedIn profile, and other useful contact information. Add this to the spreadsheet alongside the company data. So we'll run that one as well then. And we should be able to update this spreadsheet using this second connector and this second scrape. And now you'll see the beauty of co-work. We're working with that file that we've created on our computer on the right here, that spreadsheet file, and it's matching that with this new connector and this new piece of information to further enhance our prospecting campaign.
And after a few minutes, we get the result again. We've got the updated spreadsheet with the new columns. We'll see if we scrape over to the right hand side here. We've got all of the job titles, decision makers, direct email address for these people, LinkedIn profiles, and everything. And it really is as simple as that.
Now, to actually build these sorts of prospecting lists, now let's try something a bit different. Let's actually do something with some data that we're going to scrape. Now, this is quite ambitious. This is I want to be able to scrape LinkedIn job descriptions and then actually analyze them and create a PR report for them. This is something I'd do in the past all sorts of data and it could take days or weeks to actually create these reports getting all the data and analyzing it. But I'm hoping to do this really really quickly with co-work. Now this might not work. So I'm hoping it's not going to end in an epic fail for us. Let's see how we go. This is a live test.
So I've got our little prompt that we're going to use here. Again, I'm not going crazy with these prompts at the moment when I'm showing this stuff. Use Apify to scrape LinkedIn for 100 administ administrative job listings in the UK. I want rolls across different industries. Pull the job title, full job job description for each list, each listing. But we've got a bit of a mismatch in there. Let's just do it for 50 for speed sake and see if it can actually go and get all of this information from LinkedIn. Could be great if you're job hunting as well, to be honest. So, let's go with that one and see how we get on. So if we can get the data easily then we can start creating our PR reports which are like absolute gold dust for marketing link building brand building.
Okay. So it's actually gone and loaded Apify again found a scraper that can scrape LinkedIn. It's got 26,000 users and a 4.9 rating. So I'm it sounds like it should go do a good job. In the future if this works this test I could do this for thousands of job listings to actually run an analysis. I want to analyze how many job listings actually mention that they want people to know that know how to use AI that have skills in using claude chat GPT and other AI platforms to see actually are people using this as something they're recruiting for or not and create a report and analysis out the back of it that we can send off to newspapers and the press.
And to be honest, this is in less than 2 minutes. It's already scraped all of the 50 listings and is creating a spreadsheet with the job titles and full descriptions. So, it's actually working pretty dang fast to get this done. It's already done, actually. [laughter] Le definitely less than five minutes, probably about three minutes. It's actually built our spreadsheet here. Let's have a look. And we've got the full job listings, what the job is, executive assistant, office manager, the company there, and the full job description and a link to the job description as well. So, we've got this core data. So easy. That's so easy. This is insane.
Okay. Right. Let's have some more fun with this. Um, now obviously I would do this with a much larger data set in the future, but I'm just going to take this next prompt and we're going to pop it in there and analyze all the job descriptions we straight from LinkedIn. I want to know how many of them mention AI skills as a requirement. Chat, GPT, Claude, prompt engineering, AI tools, etc. Break it down. What skills are mentioned? Any patterns? Turn this into a short punchy report that could be used for a press release. Include the key stats, summary findings, and two quotable key takeaways. Frame it around the headline, X% of job listings now require AI skills. Please write the report that's from Elliot Prince AI coach and any quotes from me as well that would be useful in here.
Usually when I'm writing this report, I would pair it with some Claude skills or a particular project that knows all about my business and information, but I'm not actually going to but I don't have that set up for this cuz I didn't think to do it before. But usually I would run that in tandem with a Claude project and Claude skills to actually do this. But we'll see how this analysis goes. Get the initial insight from just these 50 job listings in the UK. And by the way, while I run this, you can see in my Apify account here, we've still only used $1.90 worth of credit. Thatif scraping 50 jobs off LinkedIn only cost 5. And we got a little teaser coming up as well.
Now I have a really clear picture. The story is actually more interesting than expected. Absence of AI requirements. Oh my god, this is actually every time I use co-work for a new thing, it blows my mind because I genuinely this would take me days to compile before. Even when I was using normal clawed chat, it would still take me ages to get the right data, go through it step by step, keep feeding it information to keep it running. Now we've got this, this is actually a crazy number. Just 2% of UK administrative jobs explicitly require AI skills, which is mindboggling for me because admin jobs are so ripe for supercharging your whole team and department, scaling without increasing headcount. Only two of them, only 2% out of the 50 we looked at. 8% mention anywhere. Probably because these businesses still don't know they exist or what they can do. But if I showed them this, I'm pretty sure they'd be converted instantly.
And then it writes our report for us. Obviously, I'd go and improve this and write it, add that human element on touch, but we've got the headline. We've got what the data shows, the charts already built, everything. And what employees are still asking for. They're looking for people who can use Microsoft Office doing Excel, PowerPoint. Well, I've just shown you that co-work can do Excel much quicker than most people can even open up a spreadsheet and get it organized. And in fact, there's irony in here in that people are asking people not to use AI, which I guess is fair enough in their applications, but don't want anything to do with it. Even it's written some quotes for me again, like I I would write these and put read the report, read all the data, double check it all and write these quotes myself rather than just allowing these offtheshelf ones that don't even match my tone of voice. And it's nicely added the methodology of how we did it at the bottom here as well. So, this is a great insight. I would level this up to maybe a,000 or 5,000 job postings now to get some really meaningful data on how many businesses are actually wanting their staff, their workers to actually know how to use AI or not. Apparently not, which is mental to