📱

Get Our Mobile App

Take your business learning on the go!

Download on the App StoreGet it on Google Play

The ChatGPT Scraping Workflow That Fixes Your AI SEO

Yaron Been15:59

Transcription

Many people are selling methods and courses and tools about AI engine optimization. Meaning how to basically make sure that your brand or website gets more mentions in results of JGBT or Antropic or Gemini or whatever. Obviously, um, there's no need to explain why this could be useful.

The thing is that, in my opinion, many of the methods or the courses are basically selling snake oil because this field is so new. Most of the people don't have solid data, and regardless, even if you had solid data, the data is somewhat contaminated because these things evolve all the time.

That being said, the new SEO game, based on my assumption and observation, and after speaking with many players in the SEO space, is kind of similar in principle to the old SEO game. If you provide high-quality content, if you have high domain authority, and if your website is mentioned in other places that are ranking high, it is more likely that you will be discovered by users.

Under this notion, I decided to try and see if I can optimize Rospig results. Raspig is a platform that I'm building. It's for creative teams that want to generate more velocity. It basically generates, analyzes, and deploys ads at scale, everything within a few clicks. And I want to get more eyeballs and I want to be discovered.

So what I did, I decided that it would make a lot of sense to see what is my current status in terms of being mentioned by Chad GPT and then look for places or posts or blog articles that already cited by CHP and try to outreach to them.

Now, initially, or what you can obviously do, you can come to CHP and just type it here. "What are the top suses for meta media buying companies?" and I can see the results. This isn't scalable and this is tedious, but it can produce results and it can give us some insights. So, triple whale, little attribution solution, motion is a creative analytic solutions for play, same, etc., etc.

But if you want to do this at scale, you need a tool that scrapes the results from CHP. You could assume that you can just use any escaper, but JGPT or OpenAI are smart enough in order to have Cloudflare and basically block bots. And this is where Bright Data comes in.

For those of you who don't know, I constantly recommend Bright Data because they are the best scrapping solution out there, in my humble opinion. They provide you the proxies, they provide you the infrastructure for solving Cloudflare, etc. Basically, you just assume that all your scraping infrastructure problems are solved by Bright Data, and in this instance, it's also true.

What you can come and do is come here to Bright Data and say, "What is your URL?" Define your prompt. So, as I said over there, "Top SAS solutions from meta media bind." You can do the same over here. In this case, let's say, "Top meta SAS solutions for media buyers." And now this is only like the playground, the UIX, the front end, and you can also define from where you're going to access. If you want to see like what pops up from the US, or what pops up from Uganda, or what pops up from Thailand. And then you can just run this manually via the playground, or you can use it with your AI agent, which is what I did.

First of all, over here you have the request, the curl request. You can also change it to Python or Ruby or whatever. And this is the code that exactly is going to trigger the same thing as you have here. As you can see, "Top meta SAS solutions." We can see it over here. Let's change this to "Top YouTubers in London in the AI space." And basically, if I click over here, "Run manually," it's going to bring me the results. And if I take this curl command, it's also going to yield the same results. Obviously, you can add many inputs. So, you can add more layers, like more things that you would like to test, whether it's in other LLMs or just more prompts.

What I did is, let me show you in open code. First of all, let me show you what my initial research showed me. So, this is the HTML generated by Claude after doing the first request. So, it says, "Rospig is currently invisible to Chach across five high-intense queries. Chach recommends Smartly.io, Magics, Reveal Bot, Add Creative, and others, but never Pig."

Now, what it did, it looked up 30 different, five different queries. And then what I asked it to do is find pages that are already ranking and look for outreach opportunities. What do I mean by that? If I know that, for example, for a certain query, for example, for a certain query which I did like "top AI solutions for marketing," what shows up or what Chach GPT is citing is this article: "Top eight creative automation tools in 2026." I know this for a fact because this is what Bright Data scraped and told me. And then we know that the website that was cited is Hunch Ads. So, what they need to do is, I can ask my agent to personalize an outreach to Hunch Ads and offer them or request them to add Rospig as a solution to the list, and or offer them money in order to have Rospig on top of the list.

So, this is the game these days in the SEO space, as far as I am aware of. People are just looking for, instead of trying to get ranked by generating your own content, go and see what is already being ranked in the LLMs. And after you saw what has been ranked in LLMs, or what you can do, you can outreach to the content editor or whatever the partnerships manager in that certain, in that specific company, and offer them money in order for them to list you in that specific blog post.

Now, let me open open code over here. So, this is the code that generated the HTML. So, here's the full plan, open in our browser. Across five test queries, JP recommends Smartly, Magic Reveal, Add Creative, etc., but never Rasping. The plan has three parts. So, first of all, we have, it's organized into different categories, like the best AI creative tools for meta, how to scale ad creative production, which is more problem-focused, comparisons category, and workflows. Then it gave me 25 outreach targets ranked by priority, as you can see. And then we can ask it, let's do a test now that we have the websites. "Can you use the Bright Data MCP ser or unlocker to find on LinkedIn relevant people that I can reach out to? For example, look in LinkedIn for Head of Content of Hunch Ads or SEO Writer in Hunch Ads."

So, let's see what it does. I'm not sure if it might take a while, but this is just like the general idea. We found via the Bright Data scraper where are the places we want to get ranked, and then we can use the agent to find the LinkedIn profiles or people that we should outreach to, or then we can obviously enrich it in Apollo or find their emails or whatever, and we can also automate the email outreach. Let's see what it does. In the meantime, let me read this. Let's go here. Let's open these websites to see what they are about. Beach Creative and Media Insights. Never heard of them. I think they are a direct competitor of Wasping. Website looks very good. Same idea. Looks very good as well. This looks very AI. Our stuff is much better. And we have an automated pipeline that generates as good-looking as we see in one click. This is nice. It's preparing the right. Let's move on to the next in the meantime. TPC.io. Nice. Let's see. It compiles all the contacts into a clean outreach list.

In the meantime, let me show you a few more things in Bright Data. Actually, I think it would be more useful instead of me just doing research about competitors while recording the video. Let me show you other scrapers that Bright Data have. They have scrapers that are already out of the box which you can use. So, for example, a TikTok scraper, a LinkedIn scraper. For example, let's go to TikTok. So, different scrapers within TikTok. So, you can scrape profiles, you can scrape posts, you can scrape TikTok shop by URL or by a certain category or by a certain keyword or by a certain shop. You can scrape comments, etc. And the HTML was automatically opened by open code. Let's see.

"People to contact for big citations. Content marketers, founders, and editors at the 16 highest priority websites that rank for your target queries. These are people who can add Rasp to the existing listical comparison posts." And we got the content marketing at Hunch with the LinkedIn profile. We got the founder at PPC.io. No need to open the profiles, but see how cool it is, how fast it was. We have prospects from all the other companies, which is very, very, very useful. And I think this is the game from now on. This is an Israeli company. It seems like it's an Israeli company by the names. Okay, cool.

Let's go back to Bright Data for a second. So, that was TikTok, and you have just a ton of scrapers available. You also have ready data sets if you don't want to scrape. And you can also build or request the Bright Data team to custom build a scraper for you if it's more complex, or you can use the Web Unlocker, which is basically the Bright Data capabilities of proxy rotation and Cloudflare bypassing in order to build your own scrippers, which is something that I most often do. I just ask Open Code, powered by my LLM, to use the Bright Data Web Unlocker to scrape a certain website. And of course, you can also use, they have an MCP and a CLI, which I use very often. Not the CLI, I mostly use the MCP just whenever I want to scrape stuff at scale during my agentic session.

I think this pretty much covers everything that I wanted to discuss today. If I had to summarize, first of all, the AI engine optimization is very new. A lot of it is snake oil these days. Many people claim they know how to crack the code, but I don't think that they really know. But on the other hand, if you operate under the assumption that SEO hasn't significantly changed and you still need to get mentioned in high-authority outlets, the flow that I just showed you, using Bright Data to scrape CHP results at scale and then using Bright Data in order to enrich the prospects and find people to reach out to, it's a flow that just makes a ton of sense. Obviously, if you have money to pay and incentivize the people that are making the decisions regarding whether or not to insert or give a citation to your brand, obviously having money is always useful. I hope you enjoyed this video. Until next time, keep on automating.